2024-09-10T17:10:34.691 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T17:10:34.704 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T17:10:34.785 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899394 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} email: skanta@redhat.com first_in_suite: false job_id: '7899394' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHDRO0xmuTpcrnU37mR453hJSH7V/GqB9M7jLyU6L18Z6b+gebNWzagLkxC7d7uN44gP2KqK3TQOoPta1XyhLxI= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T17:10:34.785 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T17:10:34.785 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T17:10:34.786 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T17:10:34.787 INFO:teuthology.task.internal:Checking packages... 2024-09-10T17:10:34.808 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T17:10:34.809 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T17:10:34.809 INFO:teuthology.packaging:ref: None 2024-09-10T17:10:34.809 INFO:teuthology.packaging:tag: None 2024-09-10T17:10:34.809 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T17:10:34.809 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T17:10:34.809 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T17:10:34.945 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858.gbad1f558 2024-09-10T17:10:34.947 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T17:10:34.952 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T17:10:34.952 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T17:10:34.958 INFO:teuthology.task.internal:Saving configuration 2024-09-10T17:10:34.966 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T17:10:34.972 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T17:10:34.996 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899394', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 17:05:25.499067', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHDRO0xmuTpcrnU37mR453hJSH7V/GqB9M7jLyU6L18Z6b+gebNWzagLkxC7d7uN44gP2KqK3TQOoPta1XyhLxI='} 2024-09-10T17:10:34.996 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T17:10:35.002 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-10T17:10:35.003 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T17:10:35.036 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0420467be0>, signals=[15]) 2024-09-10T17:10:35.037 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T17:10:35.044 INFO:teuthology.task.internal:Opening connections... 2024-09-10T17:10:35.044 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-09-10T17:10:35.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T17:10:35.122 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T17:10:35.128 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-09-10T17:10:35.203 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-09-10T17:10:35.203 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-09-10T17:10:35.261 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-09-10T17:10:35.261 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T17:10:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T17:10:35.263 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-09-10T17:10:35.281 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T17:10:35.288 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T17:10:35.292 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T17:10:35.292 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-09-10T17:10:35.318 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T17:10:35.324 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T17:10:35.324 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-09-10T17:10:35.392 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T17:10:35.421 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T17:10:35.436 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T17:10:35.436 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T17:10:35.436 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T17:10:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:10:35.453 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-09-10T17:10:35.511 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-503.el9.x86_64 2024-09-10T17:10:35.511 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-503.el9.x86_64 2024-09-10T17:10:35.511 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-09-10T17:10:39.245 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.2 MB 00:01 2024-09-10T17:10:43.946 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 20 MB 00:03 2024-09-10T17:10:48.354 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-10T17:10:49.838 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-10T17:10:50.801 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-10T17:10:56.240 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-10T17:10:58.271 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T17:10:58.272 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T17:10:58.357 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:10:58.373 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:10:58.373 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-09-10T17:10:58.373 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:10:58.373 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-09-10T17:10:58.374 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:10:58.375 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 79 M 2024-09-10T17:10:58.375 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 126 M 2024-09-10T17:10:58.375 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:10:58.544 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 193 kB/s | 23 kB 00:00 2024-09-10T17:10:58.825 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-09-10T17:10:59.183 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 44 MB/s | 29 MB 00:00 2024-09-10T17:10:59.717 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-10T17:10:59.717 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:10:59.717 INFO:teuthology.orchestra.run.smithi155.stdout:Total 59 MB/s | 79 MB 00:01 2024-09-10T17:11:00.400 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:11:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:11:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:11:01.694 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:11:01.694 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:11:04.143 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:11:04.919 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T17:11:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T17:11:07.794 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T17:11:07.927 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T17:11:12.776 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T17:11:12.994 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T17:11:17.817 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T17:11:56.134 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T17:11:56.181 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T17:11:56.578 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T17:11:56.579 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T17:11:56.579 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T17:11:56.579 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T17:11:57.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T17:11:57.381 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:11:57.381 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:11:57.381 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T17:11:57.381 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T17:11:57.382 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T17:11:57.382 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T17:11:57.382 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:11:57.382 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:11:57.726 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T17:11:57.809 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T17:11:57.810 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T17:11:57.810 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T17:11:57.810 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T17:11:57.810 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T17:11:57.810 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T17:11:57.810 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T17:11:57.810 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T17:11:57.868 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T17:11:57.868 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T17:11:57.868 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-09-10T17:11:57.900 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-09-10T17:11:57.900 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-09-10T17:11:57.900 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-09-10T17:11:57.900 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 82.el9 2024-09-10T17:11:57.900 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-09-10T17:11:57.900 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8274670 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-09-10T17:11:57.901 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-09-10T17:11:57.902 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:11:57.902 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T17:11:57.902 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T17:11:57.902 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T17:11:57.902 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-09-10T17:11:57.902 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:11:57.902 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T17:11:57.903 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T17:11:57.903 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-505.el9.x86_64 2024-09-10T17:11:57.903 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T17:11:58.576 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-09-10T17:11:59.962 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T17:11:59.982 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-09-10T17:11:59.985 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-09-10T17:12:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.dSrDA2KH6e 2024-09-10T17:12:00.001 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dSrDA2KH6e 2024-09-10T17:12:00.068 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.dSrDA2KH6e 2024-09-10T17:12:00.215 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.dSrDA2KH6e is 6KB 2024-09-10T17:12:00.264 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.dSrDA2KH6e 2024-09-10T17:12:00.278 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T17:12:00.341 INFO:teuthology.orchestra.run.smithi155.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T17:12:00.341 INFO:teuthology.orchestra.run.smithi155.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T17:12:00.341 INFO:teuthology.orchestra.run.smithi155.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T17:12:00.341 INFO:teuthology.orchestra.run.smithi155.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T17:12:00.342 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T17:12:00.477 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-09-10T17:12:30.514 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T17:12:30.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-10T17:12:30.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T17:12:48.957 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-09-10T17:12:57.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-10T17:12:57.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T17:13:01.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-09-10T17:13:13.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-10T17:13:13.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T17:13:31.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-09-10T17:13:46.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-10T17:13:46.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T17:13:47.705 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-09-10T17:13:48.128 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-10T17:13:48.128 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T17:13:48.128 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-09-10T17:13:48.145 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-505.el9.x86_64 2024-09-10T17:13:48.145 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T17:13:48.145 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T17:13:48.145 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T17:13:49.147 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T17:13:49.147 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T17:13:49.239 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-09-10T17:13:49.264 DEBUG:teuthology.parallel:result is None 2024-09-10T17:13:49.264 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T17:13:49.271 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T17:13:49.272 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T17:13:49.286 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T17:13:49.292 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T17:13:49.302 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T17:13:49.302 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T17:13:49.350 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T17:13:49.356 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T17:13:49.357 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T17:13:49.422 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:13:49.431 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:13:49.433 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T17:13:49.439 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T17:13:49.440 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T17:13:49.498 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T17:13:49.508 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T17:13:49.508 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T17:13:49.554 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T17:13:49.631 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T17:13:49.679 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T17:13:49.764 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T17:13:49.804 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-10T17:13:49.804 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T17:13:49.869 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-09-10T17:13:49.968 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T17:13:50.219 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T17:13:50.227 INFO:teuthology.task.internal:Starting timer... 2024-09-10T17:13:50.227 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T17:13:50.239 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T17:13:50.248 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-09-10T17:13:50.280 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-09-10T17:13:50.281 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T17:13:50.281 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-09-10T17:13:50.343 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-09-10T17:13:50.344 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-09-10T17:13:50.344 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T17:13:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:13:50.409 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T17:13:50.409 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-09-10T17:13:50.482 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T17:13:50.491 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T17:13:50.504 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T17:13:50.505 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T17:17:33.445 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-09-10T17:17:33.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-10T17:17:33.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T17:17:33.525 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-09-10T17:17:33.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-10T17:17:33.600 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T17:17:33.610 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T17:17:33.611 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T17:17:33.611 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T17:17:33.670 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T17:17:33.684 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T17:17:33.707 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-09-10T17:17:33.719 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-09-10T17:17:33.733 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T17:17:33.747 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T17:17:33.806 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-09-10T17:17:33.809 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T17:17:33.809 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-09-10T17:17:33.809 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T17:17:33.809 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T17:17:33.810 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T17:17:33.810 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T17:17:33.810 INFO:teuthology.run_tasks:Running task install... 2024-09-10T17:17:33.818 DEBUG:teuthology.task.install:project ceph 2024-09-10T17:17:33.818 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T17:17:33.818 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T17:17:33.818 INFO:teuthology.task.install:Using flavor: default 2024-09-10T17:17:33.824 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T17:17:33.824 INFO:teuthology.task.install:extra packages: [] 2024-09-10T17:17:33.824 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'tag': None, 'wait_for_package': False} 2024-09-10T17:17:33.825 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T17:17:33.969 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/ 2024-09-10T17:17:33.969 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4858.gbad1f558 2024-09-10T17:17:34.085 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T17:17:34.085 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-10T17:17:34.085 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T17:17:34.110 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T17:17:34.111 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T17:17:34.176 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T17:17:34.252 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T17:17:34.316 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-09-10T17:17:34.318 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-09-10T17:17:34.556 INFO:teuthology.orchestra.run.smithi155.stdout:45 files removed 2024-09-10T17:17:34.583 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-09-10T17:17:35.129 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 321 kB/s | 84 kB 00:00 2024-09-10T17:17:35.421 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-10T17:17:35.705 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-10T17:17:36.155 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.2 MB 00:00 2024-09-10T17:17:38.156 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-10T17:17:48.115 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.5 MB 00:06 2024-09-10T17:17:49.615 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 119 kB/s | 19 kB 00:00 2024-09-10T17:17:50.344 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 38 MB/s | 23 MB 00:00 2024-09-10T17:17:55.825 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-09-10T17:17:58.228 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:17:58.233 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:17:58.233 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:17:58.233 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:17:58.233 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:17:58.233 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 11 M 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.5 M 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 22 M 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 25 k 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 755 k 2024-09-10T17:17:58.234 INFO:teuthology.orchestra.run.smithi155.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 509 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 45 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 143 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T17:17:58.235 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 321 k 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 306 k 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 99 k 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout:Install 31 Packages 2024-09-10T17:17:58.236 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-09-10T17:17:58.237 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:17:58.237 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 62 M 2024-09-10T17:17:58.237 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:17:58.877 INFO:teuthology.orchestra.run.smithi155.stdout:(1/33): ceph-base-19.3.0-4858.gbad1f558.el9.x86 9.3 MB/s | 5.5 MB 00:00 2024-09-10T17:17:59.035 INFO:teuthology.orchestra.run.smithi155.stdout:(2/33): ceph-radosgw-19.3.0-4858.gbad1f558.el9. 15 MB/s | 11 MB 00:00 2024-09-10T17:17:59.060 INFO:teuthology.orchestra.run.smithi155.stdout:(3/33): ceph-selinux-19.3.0-4858.gbad1f558.el9. 137 kB/s | 25 kB 00:00 2024-09-10T17:17:59.102 INFO:teuthology.orchestra.run.smithi155.stdout:(4/33): libcephfs2-19.3.0-4858.gbad1f558.el9.x8 11 MB/s | 755 kB 00:00 2024-09-10T17:17:59.127 INFO:teuthology.orchestra.run.smithi155.stdout:(5/33): libradosstriper1-19.3.0-4858.gbad1f558. 7.4 MB/s | 509 kB 00:00 2024-09-10T17:17:59.177 INFO:teuthology.orchestra.run.smithi155.stdout:(6/33): python3-ceph-argparse-19.3.0-4858.gbad1 904 kB/s | 45 kB 00:00 2024-09-10T17:17:59.245 INFO:teuthology.orchestra.run.smithi155.stdout:(7/33): python3-ceph-common-19.3.0-4858.gbad1f5 2.1 MB/s | 143 kB 00:00 2024-09-10T17:17:59.312 INFO:teuthology.orchestra.run.smithi155.stdout:(8/33): python3-cephfs-19.3.0-4858.gbad1f558.el 2.4 MB/s | 161 kB 00:00 2024-09-10T17:17:59.412 INFO:teuthology.orchestra.run.smithi155.stdout:(9/33): librgw2-19.3.0-4858.gbad1f558.el9.x86_6 18 MB/s | 5.5 MB 00:00 2024-09-10T17:17:59.437 INFO:teuthology.orchestra.run.smithi155.stdout:(10/33): python3-rados-19.3.0-4858.gbad1f558.el 2.5 MB/s | 321 kB 00:00 2024-09-10T17:17:59.462 INFO:teuthology.orchestra.run.smithi155.stdout:(11/33): python3-rbd-19.3.0-4858.gbad1f558.el9. 6.0 MB/s | 306 kB 00:00 2024-09-10T17:17:59.487 INFO:teuthology.orchestra.run.smithi155.stdout:(12/33): python3-rgw-19.3.0-4858.gbad1f558.el9. 1.9 MB/s | 99 kB 00:00 2024-09-10T17:17:59.696 INFO:teuthology.orchestra.run.smithi155.stdout:(13/33): ceph-common-19.3.0-4858.gbad1f558.el9. 15 MB/s | 22 MB 00:01 2024-09-10T17:17:59.797 INFO:teuthology.orchestra.run.smithi155.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 320 kB/s | 107 kB 00:00 2024-09-10T17:17:59.822 INFO:teuthology.orchestra.run.smithi155.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 511 kB/s | 171 kB 00:00 2024-09-10T17:17:59.847 INFO:teuthology.orchestra.run.smithi155.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-10T17:17:59.872 INFO:teuthology.orchestra.run.smithi155.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 600 kB/s | 45 kB 00:00 2024-09-10T17:17:59.914 INFO:teuthology.orchestra.run.smithi155.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-10T17:17:59.939 INFO:teuthology.orchestra.run.smithi155.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-10T17:17:59.965 INFO:teuthology.orchestra.run.smithi155.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-10T17:17:59.990 INFO:teuthology.orchestra.run.smithi155.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 717 kB/s | 36 kB 00:00 2024-09-10T17:18:00.015 INFO:teuthology.orchestra.run.smithi155.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-10T17:18:00.040 INFO:teuthology.orchestra.run.smithi155.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-10T17:18:00.099 INFO:teuthology.orchestra.run.smithi155.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-10T17:18:00.124 INFO:teuthology.orchestra.run.smithi155.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-09-10T17:18:00.149 INFO:teuthology.orchestra.run.smithi155.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-09-10T17:18:00.166 INFO:teuthology.orchestra.run.smithi155.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-10T17:18:00.200 INFO:teuthology.orchestra.run.smithi155.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-10T17:18:00.236 INFO:teuthology.orchestra.run.smithi155.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.5 MB/s | 839 kB 00:00 2024-09-10T17:18:00.303 INFO:teuthology.orchestra.run.smithi155.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-10T17:18:00.378 INFO:teuthology.orchestra.run.smithi155.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-10T17:18:00.503 INFO:teuthology.orchestra.run.smithi155.stdout:(32/33): librados2-19.3.0-4858.gbad1f558.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-10T17:18:00.562 INFO:teuthology.orchestra.run.smithi155.stdout:(33/33): librbd1-19.3.0-4858.gbad1f558.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-10T17:18:00.567 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:18:00.567 INFO:teuthology.orchestra.run.smithi155.stdout:Total 27 MB/s | 62 MB 00:02 2024-09-10T17:18:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:18:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:18:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:18:01.400 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:18:01.401 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:18:02.942 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:18:03.024 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-10T17:18:03.059 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-10T17:18:03.084 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-10T17:18:03.106 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T17:18:03.129 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T17:18:03.333 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T17:18:03.388 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T17:18:03.796 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-10T17:18:03.869 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T17:18:03.963 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T17:18:04.052 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 8/35 2024-09-10T17:18:04.079 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T17:18:04.133 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T17:18:04.170 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-10T17:18:04.214 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 11/35 2024-09-10T17:18:04.285 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T17:18:04.314 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T17:18:04.387 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T17:18:04.422 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-10T17:18:04.543 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-10T17:18:04.573 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-10T17:18:04.615 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-10T17:18:04.662 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-10T17:18:04.691 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-10T17:18:04.733 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T17:18:04.775 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-10T17:18:05.096 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-10T17:18:05.147 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T17:18:05.246 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T17:18:05.277 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 24/35 2024-09-10T17:18:05.313 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-10T17:18:05.802 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-10T17:18:05.824 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T17:18:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T17:18:05.974 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 28/35 2024-09-10T17:18:06.005 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 29/35 2024-09-10T17:18:08.308 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T17:18:08.348 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T17:18:08.915 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T17:18:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T17:18:09.066 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T17:18:09.066 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T17:18:09.066 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T17:18:09.066 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:09.104 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T17:18:19.016 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T17:18:19.016 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-09-10T17:18:19.016 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-09-10T17:18:19.016 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-09-10T17:18:19.016 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-09-10T17:18:19.016 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-09-10T17:18:19.016 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-09-10T17:18:19.016 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-09-10T17:18:19.017 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:20.051 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T17:18:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T17:18:20.076 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T17:18:20.076 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T17:18:20.076 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T17:18:20.076 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T17:18:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:20.078 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T17:18:20.161 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T17:18:20.162 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T17:18:21.317 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T17:18:21.317 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 1/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 3/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 4/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 5/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 6/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 8/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 9/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 10/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 11/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 13/35 2024-09-10T17:18:21.318 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-10T17:18:21.319 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-10T17:18:21.320 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-10T17:18:21.320 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-10T17:18:21.320 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-10T17:18:21.320 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-10T17:18:21.320 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T17:18:21.321 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-10T17:18:21.321 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 34/35 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.166 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T17:18:22.167 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T17:18:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T17:18:22.169 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T17:18:22.169 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T17:18:22.169 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:22.169 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:18:22.497 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-09-10T17:18:23.067 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:18:23.205 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:18:23.205 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:23.205 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 50 M 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-09-10T17:18:23.206 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:23.207 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 50 M 2024-09-10T17:18:23.207 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 215 M 2024-09-10T17:18:23.207 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:18:23.509 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): libcephsqlite-19.3.0-4858.gbad1f558.el9. 799 kB/s | 161 kB 00:00 2024-09-10T17:18:23.534 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-10T17:18:23.643 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 476 kB/s | 64 kB 00:00 2024-09-10T17:18:24.826 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): ceph-test-19.3.0-4858.gbad1f558.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-10T17:18:24.828 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:18:24.828 INFO:teuthology.orchestra.run.smithi155.stdout:Total 31 MB/s | 50 MB 00:01 2024-09-10T17:18:24.856 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:18:24.876 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:18:24.877 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:18:25.255 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:18:25.255 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:18:25.794 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:18:25.870 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T17:18:25.917 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T17:18:25.945 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T17:18:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T17:18:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T17:18:33.183 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T17:18:33.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 1/4 2024-09-10T17:18:33.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2/4 2024-09-10T17:18:33.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:33.513 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:18:33.813 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-09-10T17:18:34.406 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:39 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:18:34.556 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:18:34.559 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:18:34.559 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:34.559 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:18:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 6.6 k 2024-09-10T17:18:34.559 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:18:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 2.4 M 2024-09-10T17:18:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 1.5 M 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 291 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 4.8 M 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 17 M 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T17:18:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T17:18:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout:Install 37 Packages 2024-09-10T17:18:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:34.563 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 31 M 2024-09-10T17:18:34.563 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 109 M 2024-09-10T17:18:34.563 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:18:35.295 INFO:teuthology.orchestra.run.smithi155.stdout:(1/37): ceph-19.3.0-4858.gbad1f558.el9.x86_64.r 52 kB/s | 6.6 kB 00:00 2024-09-10T17:18:35.571 INFO:teuthology.orchestra.run.smithi155.stdout:(2/37): ceph-mgr-19.3.0-4858.gbad1f558.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2024-09-10T17:18:36.046 INFO:teuthology.orchestra.run.smithi155.stdout:(3/37): ceph-mon-19.3.0-4858.gbad1f558.el9.x86_ 6.4 MB/s | 4.8 MB 00:00 2024-09-10T17:18:36.113 INFO:teuthology.orchestra.run.smithi155.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4858.gbad1 4.3 MB/s | 291 kB 00:00 2024-09-10T17:18:36.180 INFO:teuthology.orchestra.run.smithi155.stdout:(5/37): ceph-mds-19.3.0-4858.gbad1f558.el9.x86_ 2.4 MB/s | 2.4 MB 00:01 2024-09-10T17:18:36.338 INFO:teuthology.orchestra.run.smithi155.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-10T17:18:36.530 INFO:teuthology.orchestra.run.smithi155.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-09-10T17:18:36.724 INFO:teuthology.orchestra.run.smithi155.stdout:(8/37): ceph-osd-19.3.0-4858.gbad1f558.el9.x86_ 15 MB/s | 17 MB 00:01 2024-09-10T17:18:36.747 INFO:teuthology.orchestra.run.smithi155.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 623 kB/s | 135 kB 00:00 2024-09-10T17:18:36.789 INFO:teuthology.orchestra.run.smithi155.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.0 MB/s | 1.2 MB 00:00 2024-09-10T17:18:36.857 INFO:teuthology.orchestra.run.smithi155.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-09-10T17:18:36.915 INFO:teuthology.orchestra.run.smithi155.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-10T17:18:36.940 INFO:teuthology.orchestra.run.smithi155.stdout:(13/37): python3-requests-2.25.1-8.el9.noarch.r 574 kB/s | 125 kB 00:00 2024-09-10T17:18:36.966 INFO:teuthology.orchestra.run.smithi155.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 689 kB/s | 35 kB 00:00 2024-09-10T17:18:36.991 INFO:teuthology.orchestra.run.smithi155.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-10T17:18:37.016 INFO:teuthology.orchestra.run.smithi155.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-10T17:18:37.042 INFO:teuthology.orchestra.run.smithi155.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-09-10T17:18:37.067 INFO:teuthology.orchestra.run.smithi155.stdout:(18/37): python3-packaging-20.9-5.el9.noarch.rp 611 kB/s | 77 kB 00:00 2024-09-10T17:18:37.109 INFO:teuthology.orchestra.run.smithi155.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-10T17:18:37.136 INFO:teuthology.orchestra.run.smithi155.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 649 kB/s | 43 kB 00:00 2024-09-10T17:18:37.159 INFO:teuthology.orchestra.run.smithi155.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-10T17:18:37.193 INFO:teuthology.orchestra.run.smithi155.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 126 kB/s | 22 kB 00:00 2024-09-10T17:18:37.219 INFO:teuthology.orchestra.run.smithi155.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 182 kB/s | 11 kB 00:00 2024-09-10T17:18:37.252 INFO:teuthology.orchestra.run.smithi155.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 886 kB/s | 23 kB 00:00 2024-09-10T17:18:37.278 INFO:teuthology.orchestra.run.smithi155.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 835 kB/s | 19 kB 00:00 2024-09-10T17:18:37.303 INFO:teuthology.orchestra.run.smithi155.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-09-10T17:18:37.328 INFO:teuthology.orchestra.run.smithi155.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-10T17:18:37.353 INFO:teuthology.orchestra.run.smithi155.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-10T17:18:37.379 INFO:teuthology.orchestra.run.smithi155.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-10T17:18:37.404 INFO:teuthology.orchestra.run.smithi155.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-10T17:18:37.430 INFO:teuthology.orchestra.run.smithi155.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-10T17:18:37.454 INFO:teuthology.orchestra.run.smithi155.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-10T17:18:37.480 INFO:teuthology.orchestra.run.smithi155.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-10T17:18:37.513 INFO:teuthology.orchestra.run.smithi155.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-10T17:18:37.538 INFO:teuthology.orchestra.run.smithi155.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-10T17:18:37.564 INFO:teuthology.orchestra.run.smithi155.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-10T17:18:37.589 INFO:teuthology.orchestra.run.smithi155.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-10T17:18:37.592 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:18:37.593 INFO:teuthology.orchestra.run.smithi155.stdout:Total 10 MB/s | 31 MB 00:03 2024-09-10T17:18:37.743 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:18:37.783 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:18:37.784 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:18:38.279 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:18:38.279 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:18:39.627 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:18:39.700 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T17:18:39.733 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T17:18:39.769 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T17:18:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T17:18:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T17:18:39.944 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T17:18:40.037 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T17:18:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T17:18:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T17:18:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T17:18:40.236 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T17:18:40.271 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T17:18:40.304 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T17:18:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T17:18:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T17:18:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T17:18:40.483 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T17:18:40.515 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T17:18:40.588 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T17:18:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T17:18:40.816 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T17:18:40.876 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T17:18:40.915 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T17:18:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T17:18:41.028 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T17:18:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T17:18:41.269 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T17:18:41.348 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T17:18:41.436 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T17:18:41.484 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T17:18:41.624 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T17:18:41.783 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 32/37 2024-09-10T17:18:41.813 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T17:18:41.840 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T17:18:41.840 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T17:18:41.840 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T17:18:41.840 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T17:18:41.841 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T17:18:41.841 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:43.297 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T17:18:43.324 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T17:18:43.324 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T17:18:43.324 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T17:18:43.324 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T17:18:43.324 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T17:18:43.324 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:43.773 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T17:18:43.796 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T17:18:43.796 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T17:18:43.797 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T17:18:43.797 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T17:18:43.797 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T17:18:43.797 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:44.019 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T17:18:44.042 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T17:18:44.043 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T17:18:44.043 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T17:18:44.043 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T17:18:44.043 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T17:18:44.043 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:44.107 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T17:18:45.840 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 1/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 3/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 4/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 5/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 6/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T17:18:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T17:18:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T17:18:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T17:18:46.929 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T17:18:46.929 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:46.929 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:18:46.929 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:46.929 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:46.929 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:46.929 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T17:18:46.930 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T17:18:46.931 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T17:18:46.932 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T17:18:46.932 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T17:18:46.932 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T17:18:46.932 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T17:18:46.932 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T17:18:46.932 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T17:18:46.932 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:46.932 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:18:47.274 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-09-10T17:18:47.851 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:52 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:18:47.941 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:18:47.982 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:18:47.983 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:18:47.983 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:18:48.063 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-09-10T17:18:48.628 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:18:48.761 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 784 k 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 784 k 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 790 k 2024-09-10T17:18:48.762 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:18:49.018 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-19.3.0-4858.gbad1f558.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-10T17:18:49.018 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:18:49.019 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.0 MB/s | 784 kB 00:00 2024-09-10T17:18:49.019 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:18:49.024 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:18:49.025 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:18:49.030 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:18:49.031 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:18:49.102 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:18:49.596 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T17:18:49.688 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T17:18:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T17:18:50.289 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T17:18:50.289 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:50.290 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:18:50.290 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:18:50.290 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:50.290 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:18:50.529 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T17:18:51.106 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:18:51.243 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:18:51.244 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:51.244 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-09-10T17:18:51.244 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:51.245 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:18:51.245 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 145 k 2024-09-10T17:18:51.245 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:51.245 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:18:51.245 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:51.245 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-10T17:18:51.245 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:51.245 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 145 k 2024-09-10T17:18:51.246 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 432 k 2024-09-10T17:18:51.246 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:18:51.458 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-19.3.0-4858.gbad1f5 681 kB/s | 145 kB 00:00 2024-09-10T17:18:51.458 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:18:51.459 INFO:teuthology.orchestra.run.smithi155.stdout:Total 676 kB/s | 145 kB 00:00 2024-09-10T17:18:51.459 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:18:51.466 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:18:51.466 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:18:51.524 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:18:51.525 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:18:51.707 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:18:51.784 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T17:18:51.809 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T17:18:51.809 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T17:18:51.809 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T17:18:51.809 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:52.712 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T17:18:52.712 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:52.712 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:18:52.712 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:18:52.712 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:52.713 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:18:52.971 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-09-10T17:18:53.544 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:58 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:18:53.635 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:18:53.677 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:18:53.678 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:18:53.678 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:18:53.742 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T17:18:54.299 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:59 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:18:54.433 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:18:54.434 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:54.434 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:18:54.434 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:54.434 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:18:54.434 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 14 M 2024-09-10T17:18:54.434 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:18:54.434 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 34 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 16 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:18:54.435 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-09-10T17:18:54.436 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:18:54.436 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 19 M 2024-09-10T17:18:54.436 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 159 M 2024-09-10T17:18:54.436 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:18:55.022 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4858.gba 238 kB/s | 34 kB 00:00 2024-09-10T17:18:55.063 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4858.gbad 89 kB/s | 16 kB 00:00 2024-09-10T17:18:55.273 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-10T17:18:55.398 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00 2024-09-10T17:18:55.448 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-10T17:18:55.474 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-09-10T17:18:55.541 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-10T17:18:55.566 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-10T17:18:55.591 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-10T17:18:55.617 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-10T17:18:55.659 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.2 MB/s | 863 kB 00:00 2024-09-10T17:18:55.884 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4858.gbad1f5 14 MB/s | 14 MB 00:01 2024-09-10T17:18:55.887 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:18:55.887 INFO:teuthology.orchestra.run.smithi155.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-10T17:18:55.936 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:18:55.954 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:18:55.955 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:18:56.225 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:18:56.225 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:18:56.821 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:18:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T17:18:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T17:18:56.984 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T17:18:57.013 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T17:18:57.127 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T17:18:57.265 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T17:18:57.367 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T17:18:57.418 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T17:18:57.447 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T17:18:57.479 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 10/12 2024-09-10T17:19:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 11/12 2024-09-10T17:19:00.743 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T17:19:02.733 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T17:19:02.733 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 1/12 2024-09-10T17:19:02.733 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 2/12 2024-09-10T17:19:02.733 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 3/12 2024-09-10T17:19:02.733 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T17:19:02.734 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T17:19:02.734 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T17:19:02.734 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T17:19:02.734 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T17:19:02.734 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T17:19:02.734 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T17:19:02.734 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T17:19:03.052 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T17:19:03.052 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T17:19:03.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T17:19:03.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T17:19:03.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T17:19:03.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T17:19:03.054 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:03.054 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:03.316 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T17:19:03.893 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:08 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:04.035 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:04.036 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-09-10T17:19:04.036 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 7.4 M 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T17:19:04.037 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T17:19:04.038 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T17:19:04.038 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:04.038 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:04.038 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-09-10T17:19:04.038 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-09-10T17:19:04.038 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:04.038 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 38 M 2024-09-10T17:19:04.038 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 212 M 2024-09-10T17:19:04.039 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:04.451 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-10T17:19:04.493 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-10T17:19:04.568 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-10T17:19:04.619 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-10T17:19:04.694 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 513 kB/s | 38 kB 00:00 2024-09-10T17:19:04.828 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-48 14 MB/s | 7.4 MB 00:00 2024-09-10T17:19:05.003 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-10T17:19:05.436 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.2 MB/s | 3.0 MB 00:00 2024-09-10T17:19:05.545 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-10T17:19:06.321 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.0 MB/s | 4.9 MB 00:01 2024-09-10T17:19:06.404 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.6 MB/s | 5.1 MB 00:01 2024-09-10T17:19:08.671 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.2 MB/s | 16 MB 00:03 2024-09-10T17:19:08.671 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:08.672 INFO:teuthology.orchestra.run.smithi155.stdout:Total 8.2 MB/s | 38 MB 00:04 2024-09-10T17:19:09.030 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:09.057 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:09.057 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:09.474 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:09.474 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:10.237 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:10.308 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T17:19:10.352 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T17:19:10.451 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T17:19:10.496 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T17:19:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T17:19:10.711 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T17:19:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T17:19:11.445 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T17:19:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T17:19:12.819 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T17:19:13.861 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T17:19:13.886 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T17:19:16.435 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 1/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T17:19:16.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T17:19:16.859 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T17:19:16.859 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T17:19:16.860 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T17:19:16.861 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T17:19:16.861 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:16.861 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:17.221 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-09-10T17:19:17.802 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:22 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:17.938 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 49 k 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T17:19:17.939 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-09-10T17:19:17.940 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:17.941 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.2 M 2024-09-10T17:19:17.941 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-09-10T17:19:17.941 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:18.346 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): ceph-mgr-rook-19.3.0-4858.gbad1f558.el9 301 kB/s | 49 kB 00:00 2024-09-10T17:19:18.697 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-10T17:19:18.747 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 45 kB/s | 26 kB 00:00 2024-09-10T17:19:19.197 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 261 kB/s | 222 kB 00:00 2024-09-10T17:19:19.414 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 221 kB/s | 159 kB 00:00 2024-09-10T17:19:19.464 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 762 kB/s | 32 kB 00:00 2024-09-10T17:19:19.490 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 563 kB/s | 14 kB 00:00 2024-09-10T17:19:19.514 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.7 MB/s | 220 kB 00:00 2024-09-10T17:19:19.565 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 21 MB/s | 1.0 MB 00:00 2024-09-10T17:19:19.590 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-10T17:19:19.615 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 321 kB/s | 279 kB 00:00 2024-09-10T17:19:19.641 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-10T17:19:19.849 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 82 kB/s | 54 kB 00:00 2024-09-10T17:19:19.851 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:19.852 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-10T17:19:19.922 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:19.938 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:19.938 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:20.028 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:20.029 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:20.539 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:20.707 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T17:19:20.767 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T17:19:20.831 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T17:19:20.869 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T17:19:20.910 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T17:19:21.012 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T17:19:21.114 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T17:19:21.173 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T17:19:21.881 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T17:19:21.951 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T17:19:21.985 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T17:19:22.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T17:19:22.054 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T17:19:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T17:19:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 1/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T17:19:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T17:19:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T17:19:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T17:19:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T17:19:23.174 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T17:19:23.174 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T17:19:23.174 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T17:19:23.174 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T17:19:23.174 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:23.174 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:23.431 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T17:19:24.001 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:28 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:24.134 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:24.135 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:24.135 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:19:24.135 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 154 k 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-09-10T17:19:24.136 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:24.137 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 7.0 M 2024-09-10T17:19:24.137 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 33 M 2024-09-10T17:19:24.137 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:24.918 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-10T17:19:24.969 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-10T17:19:24.994 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4858.gbad1f558.e 536 kB/s | 154 kB 00:00 2024-09-10T17:19:25.020 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-10T17:19:25.062 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-10T17:19:26.462 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2024-09-10T17:19:26.465 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:26.465 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.0 MB/s | 7.0 MB 00:02 2024-09-10T17:19:26.540 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:26.551 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:26.551 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:26.652 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:26.653 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:26.966 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:27.137 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T17:19:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T17:19:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T17:19:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T17:19:27.889 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T17:19:27.918 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T17:19:28.638 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T17:19:28.638 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/6 2024-09-10T17:19:28.638 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T17:19:28.639 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T17:19:28.639 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T17:19:28.639 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T17:19:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T17:19:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:28.914 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:19:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T17:19:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T17:19:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T17:19:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T17:19:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T17:19:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:28.915 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:29.201 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-09-10T17:19:29.765 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:29.897 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:29.897 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 901 k 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-09-10T17:19:29.898 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:29.899 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 981 k 2024-09-10T17:19:29.899 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.9 M 2024-09-10T17:19:29.899 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:30.039 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 669 kB/s | 80 kB 00:00 2024-09-10T17:19:30.206 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-19.3.0-4858.gbad1f558.el9.x86_ 3.1 MB/s | 901 kB 00:00 2024-09-10T17:19:30.206 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:30.207 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.1 MB/s | 981 kB 00:00 2024-09-10T17:19:30.218 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:30.240 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:30.240 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:30.320 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:30.320 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:30.509 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T17:19:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T17:19:31.363 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T17:19:31.363 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/2 2024-09-10T17:19:31.663 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T17:19:31.664 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:31.664 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:31.664 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T17:19:31.664 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:31.664 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:31.947 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-volume 2024-09-10T17:19:32.533 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:32.674 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:32.674 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:32.674 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-10T17:19:32.674 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 285 k 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 285 k 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.3 M 2024-09-10T17:19:32.675 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:32.920 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-volume-19.3.0-4858.gbad1f558.el9.noarch.rp 1.1 MB/s | 285 kB 00:00 2024-09-10T17:19:32.920 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:32.920 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-10T17:19:32.921 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:32.929 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:32.929 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:32.956 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:32.956 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:33.177 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:33.247 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T17:19:33.263 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T17:19:33.263 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T17:19:33.263 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T17:19:33.264 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:34.273 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T17:19:34.273 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:34.273 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:34.273 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T17:19:34.273 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:34.273 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:34.561 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-09-10T17:19:35.132 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:35.270 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:35.271 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:35.271 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-09-10T17:19:35.271 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:35.271 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:35.271 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 126 k 2024-09-10T17:19:35.272 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:35.272 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:35.272 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:35.272 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-10T17:19:35.272 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:35.272 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 126 k 2024-09-10T17:19:35.272 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 456 k 2024-09-10T17:19:35.272 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:35.484 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-19.3.0-4858.gbad1f558.el9.x86_64 599 kB/s | 126 kB 00:00 2024-09-10T17:19:35.484 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:35.484 INFO:teuthology.orchestra.run.smithi155.stdout:Total 594 kB/s | 126 kB 00:00 2024-09-10T17:19:35.485 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:35.490 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:35.490 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:35.554 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:35.554 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:35.699 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:35.789 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:36.257 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:36.515 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:19:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:36.515 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:36.777 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-09-10T17:19:37.351 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:37.444 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:19:37.490 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:37.490 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:19:37.491 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:37.565 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-09-10T17:19:38.131 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:38.264 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:38.265 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:38.265 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-09-10T17:19:38.265 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:38.265 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 32 k 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 32 k 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 142 k 2024-09-10T17:19:38.266 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:38.421 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-19.3.0-4858.gbad1f558.el9.x86_6 206 kB/s | 32 kB 00:00 2024-09-10T17:19:38.421 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:38.422 INFO:teuthology.orchestra.run.smithi155.stdout:Total 203 kB/s | 32 kB 00:00 2024-09-10T17:19:38.422 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:38.425 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:38.425 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:38.456 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:38.456 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:38.548 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:38.667 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:39.035 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:39.269 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:19:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:39.269 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:39.434 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-09-10T17:19:40.003 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:40.093 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:19:40.136 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:40.137 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:19:40.137 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:40.210 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-09-10T17:19:40.775 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:40.866 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:19:40.909 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:40.910 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:19:40.910 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:40.977 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-09-10T17:19:41.544 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:46 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:41.636 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:19:41.680 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:41.681 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:19:41.681 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:41.749 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-09-10T17:19:42.311 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:47 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:42.402 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:19:42.445 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:42.446 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:19:42.446 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:42.517 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-09-10T17:19:43.088 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:48 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:43.179 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:19:43.223 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:43.224 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:19:43.224 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:43.295 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-09-10T17:19:43.862 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:48 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:43.952 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T17:19:43.996 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:43.997 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-10T17:19:43.997 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:44.070 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-09-10T17:19:44.639 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:49 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:44.771 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:44.771 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:44.771 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-09-10T17:19:44.771 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 85 k 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 85 k 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 231 k 2024-09-10T17:19:44.772 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:44.974 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-19.3.0-4858.gbad1f558.el9.x86_64.rpm 423 kB/s | 85 kB 00:00 2024-09-10T17:19:44.974 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:44.974 INFO:teuthology.orchestra.run.smithi155.stdout:Total 420 kB/s | 85 kB 00:00 2024-09-10T17:19:44.975 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:44.981 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:44.981 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:45.045 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:45.046 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:45.193 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:45.279 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:45.643 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:45.890 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:19:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:45.891 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:46.134 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-09-10T17:19:46.710 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:51 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:46.846 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:46.847 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:46.847 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-09-10T17:19:46.847 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:46.847 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:46.847 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T17:19:46.848 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:46.848 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:46.848 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:46.848 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-10T17:19:46.848 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:46.848 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.2 M 2024-09-10T17:19:46.848 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 14 M 2024-09-10T17:19:46.848 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:47.186 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-19.3.0-4858.gbad1f558.el9.x86_64.rpm 9.4 MB/s | 3.2 MB 00:00 2024-09-10T17:19:47.187 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:47.187 INFO:teuthology.orchestra.run.smithi155.stdout:Total 9.4 MB/s | 3.2 MB 00:00 2024-09-10T17:19:47.187 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:47.197 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:47.197 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:47.284 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:47.284 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:47.721 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:47.779 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:47.803 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:47.803 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T17:19:47.803 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T17:19:47.804 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T17:19:47.804 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T17:19:47.804 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:48.903 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:48.903 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:48.903 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:48.903 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:19:48.904 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:48.904 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:49.169 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-09-10T17:19:49.757 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:54 ago on Tue 10 Sep 2024 05:17:55 PM UTC. 2024-09-10T17:19:49.891 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-10T17:19:49.891 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:49.891 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 170 k 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 170 k 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 490 k 2024-09-10T17:19:49.892 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-10T17:19:50.082 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-19.3.0-4858.gbad1f558.el9.x86_64.rpm 899 kB/s | 170 kB 00:00 2024-09-10T17:19:50.083 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-10T17:19:50.083 INFO:teuthology.orchestra.run.smithi155.stdout:Total 893 kB/s | 170 kB 00:00 2024-09-10T17:19:50.083 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-10T17:19:50.090 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-10T17:19:50.090 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-10T17:19:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-10T17:19:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-10T17:19:50.309 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-10T17:19:50.429 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:50.839 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:51.098 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T17:19:51.098 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:51.098 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-10T17:19:51.098 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T17:19:51.098 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-10T17:19:51.098 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-10T17:19:51.344 DEBUG:teuthology.parallel:result is None 2024-09-10T17:19:51.345 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T17:19:51.489 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T17:19:51.511 INFO:teuthology.orchestra.run.smithi155.stdout:19.3.0-4858.gbad1f558.el9 2024-09-10T17:19:51.512 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858.gbad1f558.el9 2024-09-10T17:19:51.512 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858.gbad1f558 is installed. 2024-09-10T17:19:51.514 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T17:19:51.515 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-10T17:19:51.515 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T17:19:51.577 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T17:19:51.578 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-10T17:19:51.578 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T17:19:51.642 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T17:19:51.707 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T17:19:51.708 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-10T17:19:51.708 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T17:19:51.772 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T17:19:51.836 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T17:19:51.836 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-10T17:19:51.836 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T17:19:51.901 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T17:19:51.965 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T17:19:51.979 INFO:tasks.workunit:Pulling workunits from ref bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T17:19:51.980 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T17:19:51.980 INFO:tasks.workunit:timeout=3h 2024-09-10T17:19:51.980 INFO:tasks.workunit:cleanup=True 2024-09-10T17:19:51.980 DEBUG:teuthology.orchestra.run.smithi155:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T17:19:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:19:52.021 INFO:teuthology.orchestra.run.smithi155.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T17:19:52.021 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T17:19:52.076 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T17:19:52.076 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T17:19:52.132 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T17:19:52.187 INFO:tasks.workunit.client.0.smithi155.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T17:20:41.783 INFO:tasks.workunit.client.0.smithi155.stderr:Note: switching to 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'. 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-10T17:20:41.784 INFO:tasks.workunit.client.0.smithi155.stderr: git switch -c 2024-09-10T17:20:41.785 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-10T17:20:41.785 INFO:tasks.workunit.client.0.smithi155.stderr:Or undo this operation with: 2024-09-10T17:20:41.785 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-10T17:20:41.785 INFO:tasks.workunit.client.0.smithi155.stderr: git switch - 2024-09-10T17:20:41.785 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-10T17:20:41.785 INFO:tasks.workunit.client.0.smithi155.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T17:20:41.785 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-10T17:20:41.785 INFO:tasks.workunit.client.0.smithi155.stderr:HEAD is now at bad1f558c32 Merge branch 'wip-ifed-bluestore-tool-really-really' of https://github.com/ifed01/ceph into wip-bharath7-testing-2024-09-10-1409 2024-09-10T17:20:41.787 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T17:20:41.844 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-10T17:20:41.844 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T17:20:41.900 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-09-10T17:20:41.901 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-09-10T17:20:41.901 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-09-10T17:20:41.966 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T17:20:41.972 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T17:20:41.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:20:41.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:20:41.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T17:20:41.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T17:20:41.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/mclock-config 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-09-10T17:20:41.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-09-10T17:20:41.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none ' 2024-09-10T17:20:41.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-10T17:20:41.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-09-10T17:20:41.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-09-10T17:20:41.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T17:20:41.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_profile_builtin_to_custom 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_profile_custom_to_builtin 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_profile_disallow_builtin_params_modify 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_profile_disallow_builtin_params_override 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_recovery_limit_adjustment_mclock' 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:20:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:20:41.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:20:41.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:20:41.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:20:41.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:20:41.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:20:41.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:20:41.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:20:41.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:20:41.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:20:41.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:20:41.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:20:41.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:20:41.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:20:41.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:20:41.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:20:41.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:20:41.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:20:41.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:20:41.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:20:41.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:20:41.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:41.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:41.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:20:41.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:20:41.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:20:41.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:20:41.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:20:41.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:41.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:41.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:20:41.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:20:41.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:20:41.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:20:41.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:20:41.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:20:41.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:20:41.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:20:41.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:20:41.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:20:41.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:20:41.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:20:41.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:20:41.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:20:41.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:20:41.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:20:41.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:20:41.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:20:41.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:20:41.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:20:41.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:20:41.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:20:41.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:20:41.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:20:41.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:41.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:41.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:20:41.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:20:41.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:20:41.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:20:41.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:20:41.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:41.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:41.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:20:42.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:20:42.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-10T17:20:42.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-10T17:20:42.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:20:42.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:20:42.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:20:42.557 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:20:42.557 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:42.558 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:42.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:20:42.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:20:42.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:20:42.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:20:42.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:20:42.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:20:42.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:20:42.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:20:42.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:20:42.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:20:42.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:20:42.744 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:20:42.744 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:42.744 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:42.744 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:20:42.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:20:42.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get fsid 2024-09-10T17:20:42.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:20:42.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:20:42.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:20:42.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:20:42.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:20:42.828 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:20:42.828 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:20:42.829 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:20:42.829 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:20:42.829 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:42.829 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:42.829 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:20:42.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:20:42.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get mon_host 2024-09-10T17:20:42.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-10T17:20:42.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-10T17:20:42.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:20:42.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:20:42.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:20:42.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-10T17:20:42.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:20:43.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:20:43.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:20:43.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:20:43.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:20:43.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:43.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:43.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:20:43.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:20:43.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:20:43.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-10T17:20:43.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-10T17:20:43.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:20:43.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:20:43.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:20:43.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:20:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:20:43.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:20:43.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:20:43.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-10T17:20:43.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-10T17:20:43.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:20:43.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=13e6a3c1-6891-4791-98b5-cd01a4318d49 2024-09-10T17:20:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 13e6a3c1-6891-4791-98b5-cd01a4318d49' 2024-09-10T17:20:43.781 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 13e6a3c1-6891-4791-98b5-cd01a4318d49 2024-09-10T17:20:43.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:20:43.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDrf+BmVBZSLxAA9qdxjjiR6dHQ4GUt7AHxaw== 2024-09-10T17:20:43.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDrf+BmVBZSLxAA9qdxjjiR6dHQ4GUt7AHxaw=="}' 2024-09-10T17:20:43.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 13e6a3c1-6891-4791-98b5-cd01a4318d49 -i td/mclock-config/0/new.json 2024-09-10T17:20:44.092 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:20:44.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-10T17:20:44.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDrf+BmVBZSLxAA9qdxjjiR6dHQ4GUt7AHxaw== --osd-uuid 13e6a3c1-6891-4791-98b5-cd01a4318d49 2024-09-10T17:20:44.166 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:44.163+0000 7fc8fb934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:44.169 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:44.167+0000 7fc8fb934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:44.170 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:44.168+0000 7fc8fb934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:44.170 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:44.169+0000 7fc8fb934640 -1 bdev(0x56270be4bc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:20:44.170 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:44.169+0000 7fc8fb934640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-10T17:20:50.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-10T17:20:50.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:20:50.428 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:20:50.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:20:50.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:20:50.963 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:20:50.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:20:50.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-10T17:20:50.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:20:50.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:20:50.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:20:50.989 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:50.988+0000 7f8d13411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:51.002 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:51.000+0000 7f8d13411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:51.003 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:51.002+0000 7f8d13411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:20:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:20:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:20:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:20:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:20:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:20:51.390 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:20:51.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:51.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:20:51.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:52.117 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:52.116+0000 7f8d13411640 -1 Falling back to public interface 2024-09-10T17:20:52.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:52.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:52.781 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:20:52.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:20:52.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:52.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:20:53.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:53.588 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:20:53.586+0000 7f8d13411640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:20:54.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:54.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:54.172 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:20:54.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:20:54.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:54.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:20:54.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:55.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:55.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:55.650 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:20:55.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:20:55.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:55.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:20:56.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:57.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:57.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:57.040 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:20:57.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:20:57.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:57.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:20:57.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:58.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:58.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:58.453 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:20:58.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:20:58.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:58.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:20:58.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:59.868 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:20:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:20:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:00.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:01.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:01.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:01.342 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:21:01.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:21:01.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:01.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:01.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:02.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:02.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:02.738 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:21:02.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:21:02.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:02.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:03.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:04.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:04.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:04.156 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:21:04.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:21:04.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:04.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:04.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:05.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:05.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:05.932 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:21:05.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:21:05.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:05.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:06.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:07.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:07.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:07.567 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:21:07.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:21:07.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:07.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:08.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:09.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:09.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:09.020 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:21:09.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:21:09.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:09.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:09.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:10.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:10.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:10.593 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:21:10.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:21:10.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:10.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:11.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:12.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:12.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:12.400 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:21:12.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:21:12.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:12.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:13.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:14.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:14.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:14.284 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:21:14.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:21:14.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:14.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:15.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:16.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:16.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:16.128 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:21:16.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:21:16.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:16.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:17.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:18.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:18.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:18.684 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:21:18.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:21:18.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:18.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:19.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:20.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:20.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:20.115 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-10T17:21:20.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:21:20.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:20.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:20.546 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/440614496,v1:127.0.0.1:6803/440614496] [v2:127.0.0.1:6804/440614496,v1:127.0.0.1:6805/440614496] exists,up 13e6a3c1-6891-4791-98b5-cd01a4318d49 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:20.548 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:20.549 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:20.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:20.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:20.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_max_backfills 2024-09-10T17:21:20.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-09-10T17:21:20.632 INFO:tasks.workunit.client.0.smithi155.stdout:osd_max_backfills: 1 2024-09-10T17:21:20.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-09-10T17:21:20.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-10T17:21:21.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-10T17:21:23.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-10T17:21:23.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:23.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:23.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:23.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:23.177 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:23.177 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:23.177 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:23.177 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:23.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:23.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_max_backfills 2024-09-10T17:21:23.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-09-10T17:21:23.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:23.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:23.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok dump_recovery_reservations 2024-09-10T17:21:23.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:23.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok dump_recovery_reservations 2024-09-10T17:21:23.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-10T17:21:23.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-10T17:21:23.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-10T17:21:23.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-10T17:21:24.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-10T17:21:26.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-10T17:21:26.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:26.361 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:26.362 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:26.362 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:26.362 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:26.362 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:26.362 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:26.362 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:26.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:26.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_max_backfills 2024-09-10T17:21:26.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-10T17:21:26.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-10T17:21:26.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-10T17:21:26.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:26.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:26.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:26.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:26.458 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:26.458 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:26.458 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:26.459 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:26.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:26.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok dump_recovery_reservations 2024-09-10T17:21:26.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-10T17:21:26.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:26.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok dump_recovery_reservations 2024-09-10T17:21:26.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-10T17:21:26.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-10T17:21:26.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-09-10T17:21:26.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:21:26.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:21:26.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:21:26.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:21:26.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:21:26.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:21:26.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-09-10T17:21:27.366 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-10T17:21:27.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-09-10T17:21:27.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2024-09-10T17:21:27.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:21:27.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:21:27.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:21:27.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:27.380 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:21:27.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:21:27.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:27.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2024-09-10T17:21:27.781 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/440614496,v1:127.0.0.1:6803/440614496] [v2:127.0.0.1:6804/440614496,v1:127.0.0.1:6805/440614496] exists 13e6a3c1-6891-4791-98b5-cd01a4318d49 2024-09-10T17:21:27.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:21:27.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:21:27.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/mclock-config 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/mclock-config/0 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:21:27.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:27.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-09-10T17:21:27.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/mclock-config/0 2024-09-10T17:21:27.789 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:21:27.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T17:21:27.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-09-10T17:21:27.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/mclock-config/0/whoami 2024-09-10T17:21:27.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T17:21:27.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:21:27.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:21:27.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:21:27.811 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:27.810+0000 7f0d01a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:27.813 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:27.812+0000 7f0d01a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:27.814 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:27.813+0000 7f0d01a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:28.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T17:21:28.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:21:28.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:21:28.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:21:28.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:21:28.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:28.209 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:21:28.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:21:28.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:28.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:28.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:28.909 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:28.907+0000 7f0d01a0c640 -1 Falling back to public interface 2024-09-10T17:21:29.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:29.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:29.591 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:21:29.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:21:29.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:29.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:29.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:30.810 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:30.808+0000 7f0d01a0c640 -1 osd.0 6 log_to_monitors true 2024-09-10T17:21:30.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:30.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:30.989 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:21:30.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:21:30.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:30.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:31.491 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/842309580,v1:127.0.0.1:6803/842309580] [v2:127.0.0.1:6804/842309580,v1:127.0.0.1:6805/842309580] exists,up 13e6a3c1-6891-4791-98b5-cd01a4318d49 2024-09-10T17:21:31.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:21:31.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:21:31.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:21:31.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-10T17:21:31.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:31.493 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:31.493 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:31.493 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:31.494 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:31.494 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:31.494 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:31.494 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:31.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:31.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_max_backfills 2024-09-10T17:21:31.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-10T17:21:31.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-10T17:21:31.579 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-10T17:21:31.579 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:31.579 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:31.579 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:31.579 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:31.579 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:31.579 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:31.579 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:31.580 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:31.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:31.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok dump_recovery_reservations 2024-09-10T17:21:31.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-10T17:21:31.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-10T17:21:31.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-10T17:21:31.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-09-10T17:21:31.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:21:31.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:21:31.668 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:21:31.668 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:31.668 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:31.668 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:31.668 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:21:31.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:21:31.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok dump_recovery_reservations 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:21:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:21:31.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:21:31.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:21:31.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:21:31.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:21:31.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:21:31.861 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:21:31.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:21:31.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:21:31.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:21:31.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:21:31.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:21:31.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:21:31.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:21:31.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:21:31.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:21:31.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:21:31.870 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:21:31.870 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:31.870 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:31.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:21:31.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:21:31.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:21:31.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:21:31.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:21:31.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:21:31.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:21:31.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:21:31.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:21:31.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:21:31.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:21:31.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:21:31.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:21:31.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:21:31.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:21:31.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:21:31.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:21:31.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:31.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:31.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:21:31.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:21:31.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:21:31.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:21:31.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:21:31.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:21:31.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:21:31.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:21:31.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:21:31.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:21:31.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:21:31.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:21:31.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:21:31.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:21:31.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:21:31.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:21:31.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:21:31.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:21:31.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:31.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:31.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:21:31.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:21:31.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:21:31.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:21:31.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:21:31.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:31.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:31.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:21:31.893 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:21:31.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-10T17:21:31.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-10T17:21:32.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:21:32.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:21:32.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:21:32.354 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:21:32.354 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:32.354 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:32.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:21:32.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:21:32.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:21:32.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:21:32.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:21:32.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:21:32.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:21:32.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:21:32.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:21:32.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:21:32.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:21:32.548 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:32.548 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:32.548 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:32.548 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:21:32.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:21:32.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get fsid 2024-09-10T17:21:32.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:21:32.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:21:32.630 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get mon_host 2024-09-10T17:21:32.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-09-10T17:21:32.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-10T17:21:32.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:21:32.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:21:32.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:21:32.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-10T17:21:32.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:21:33.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:21:33.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:21:33.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:21:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:21:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:33.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:21:33.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:21:33.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:21:33.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-10T17:21:33.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-10T17:21:33.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:21:33.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:21:33.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:21:33.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:21:33.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=072e5fea-16e3-4bab-b44c-3f7ae6614cbf 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 072e5fea-16e3-4bab-b44c-3f7ae6614cbf' 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 072e5fea-16e3-4bab-b44c-3f7ae6614cbf 2024-09-10T17:21:33.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:21:33.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdgOBmfRGaIxAAALPLN2O/kfkY0tnkkqQJQQ== 2024-09-10T17:21:33.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdgOBmfRGaIxAAALPLN2O/kfkY0tnkkqQJQQ=="}' 2024-09-10T17:21:33.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 072e5fea-16e3-4bab-b44c-3f7ae6614cbf -i td/mclock-config/0/new.json 2024-09-10T17:21:33.854 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:21:33.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-10T17:21:33.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAdgOBmfRGaIxAAALPLN2O/kfkY0tnkkqQJQQ== --osd-uuid 072e5fea-16e3-4bab-b44c-3f7ae6614cbf 2024-09-10T17:21:33.893 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:33.891+0000 7f03b681e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:33.914 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:33.913+0000 7f03b681e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:33.915 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:33.914+0000 7f03b681e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:33.916 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:33.914+0000 7f03b681e640 -1 bdev(0x555ba029dc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:21:33.916 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:33.914+0000 7f03b681e640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-10T17:21:39.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-10T17:21:39.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:21:39.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:21:39.652 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:21:39.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:21:40.182 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:21:40.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:21:40.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-10T17:21:40.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:21:40.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:21:40.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:21:40.209 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:40.207+0000 7f22b840b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:40.227 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:40.225+0000 7f22b840b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:40.228 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:40.227+0000 7f22b840b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:21:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:21:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:21:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:21:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:21:40.625 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:21:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:21:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:40.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:41.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:41.824 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:41.822+0000 7f22b840b640 -1 Falling back to public interface 2024-09-10T17:21:42.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:42.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:42.013 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:21:42.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:21:42.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:42.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:42.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:43.395 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:21:43.394+0000 7f22b840b640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:21:43.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:43.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:43.982 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:21:43.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:21:43.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:43.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:43.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:44.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:44.999 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:21:44.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:45.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:21:45.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:45.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:45.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:46.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:46.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:46.428 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:21:46.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:21:46.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:46.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:46.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:47.859 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:21:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:21:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:48.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:49.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:49.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:49.273 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:21:49.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:21:49.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:49.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:49.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:50.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:50.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:50.695 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:21:50.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:21:50.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:50.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:51.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:52.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:52.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:52.163 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:21:52.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:21:52.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:52.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:52.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:53.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:53.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:53.591 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:21:53.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:21:53.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:53.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:54.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:55.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:55.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:55.065 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:21:55.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:21:55.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:55.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:55.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:56.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:56.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:56.950 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:21:56.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:21:56.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:56.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:57.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:58.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:58.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:58.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:21:58.442 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:21:58.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:58.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:21:59.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:00.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:00.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:00.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:22:00.665 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:22:00.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:00.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:01.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:02.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:02.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:02.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:22:02.550 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:22:02.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:02.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:03.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:04.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:04.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:04.085 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:22:04.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:22:04.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:04.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:05.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:06.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:06.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:06.003 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:22:06.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:22:06.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:06.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:06.398 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2765238627,v1:127.0.0.1:6803/2765238627] [v2:127.0.0.1:6804/2765238627,v1:127.0.0.1:6805/2765238627] exists,up 072e5fea-16e3-4bab-b44c-3f7ae6614cbf 2024-09-10T17:22:06.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:22:06.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:22:06.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:22:06.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-09-10T17:22:06.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-09-10T17:22:06.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:06.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-10T17:22:06.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_profile 2024-09-10T17:22:06.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-09-10T17:22:06.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-09-10T17:22:06.868 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-09-10T17:22:06.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-09-10T17:22:06.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-09-10T17:22:06.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-10T17:22:06.971 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:22:06.971 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "" 2024-09-10T17:22:06.971 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:22:06.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-10T17:22:06.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-10T17:22:06.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:06.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:06.982 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:06.982 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:06.982 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:06.982 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:06.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-10T17:22:06.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_profile 2024-09-10T17:22:07.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-09-10T17:22:07.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-09-10T17:22:07.065 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-09-10T17:22:07.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-09-10T17:22:07.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:07.067 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:07.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-10T17:22:07.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:22:07.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-09-10T17:22:07.150 INFO:tasks.workunit.client.0.smithi155.stdout:client_res = 0.300000 2024-09-10T17:22:07.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-09-10T17:22:07.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-09-10T17:22:07.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-09-10T17:22:07.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-09-10T17:22:07.151 INFO:tasks.workunit.client.0.smithi155.stdout:client_res_new = .400000 2024-09-10T17:22:07.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-09-10T17:22:07.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-10T17:22:07.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-10T17:22:08.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-09-10T17:22:08.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-10T17:22:08.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-09-10T17:22:08.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-10T17:22:08.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-10T17:22:08.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-09-10T17:22:08.023 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-10T17:22:08.023 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:08.023 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:08.023 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:08.023 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.024 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:08.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-10T17:22:08.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:22:08.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-09-10T17:22:08.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-10T17:22:08.110 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-09-10T17:22:08.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-10T17:22:08.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-09-10T17:22:08.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:22:08.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:22:08.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:22:08.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:22:08.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:22:08.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:22:08.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:22:08.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:22:08.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:22:08.218 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:22:08.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:22:08.218 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:22:08.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:22:08.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:22:08.219 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:22:08.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:08.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:22:08.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:22:08.221 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:08.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:22:08.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:22:08.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:22:08.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:22:08.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:22:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:22:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:22:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:22:08.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:22:08.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:22:08.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:22:08.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:22:08.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:22:08.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:22:08.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:22:08.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:08.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:22:08.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:22:08.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:08.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:22:08.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:22:08.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:22:08.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:22:08.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:22:08.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:22:08.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:22:08.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:22:08.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:22:08.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:22:08.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:22:08.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:22:08.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:22:08.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:22:08.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:08.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:22:08.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:22:08.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:08.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:22:08.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:22:08.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:22:08.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:22:08.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:22:08.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:22:08.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:22:08.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:22:08.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:22:08.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:22:08.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:22:08.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:22:08.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:22:08.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:22:08.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:22:08.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:22:08.253 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:22:08.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:22:08.253 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:22:08.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:22:08.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:22:08.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:22:08.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:08.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:22:08.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:22:08.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:08.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:22:08.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:22:08.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:22:08.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:22:08.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:22:08.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:22:08.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:22:08.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:22:08.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:22:08.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:22:08.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:22:08.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-10T17:22:08.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-10T17:22:08.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:22:08.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:22:08.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:22:08.659 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:22:08.659 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.659 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:22:08.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:22:08.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:22:08.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:22:08.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:22:08.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:22:08.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:22:08.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:22:08.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get fsid 2024-09-10T17:22:08.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:22:08.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get mon_host 2024-09-10T17:22:08.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-09-10T17:22:08.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-10T17:22:08.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:22:08.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:22:08.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:22:08.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-10T17:22:08.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:22:09.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:22:09.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:22:09.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:22:09.837 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:22:09.837 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:09.838 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:09.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:22:09.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:22:09.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:22:09.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-10T17:22:09.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-10T17:22:09.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:22:09.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:22:09.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:22:09.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-10T17:22:09.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-10T17:22:09.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:22:09.868 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:22:09.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-10T17:22:09.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-10T17:22:09.870 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:22:09.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1e9e98fe-77f1-4c08-a2d4-95b4c1a223fe 2024-09-10T17:22:09.870 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 1e9e98fe-77f1-4c08-a2d4-95b4c1a223fe 2024-09-10T17:22:09.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1e9e98fe-77f1-4c08-a2d4-95b4c1a223fe' 2024-09-10T17:22:09.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:22:09.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBgOBmAqetNBAAdx0BwudPWILLxasK4gWzRQ== 2024-09-10T17:22:09.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBgOBmAqetNBAAdx0BwudPWILLxasK4gWzRQ=="}' 2024-09-10T17:22:09.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1e9e98fe-77f1-4c08-a2d4-95b4c1a223fe -i td/mclock-config/0/new.json 2024-09-10T17:22:10.140 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:22:10.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-10T17:22:10.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBBgOBmAqetNBAAdx0BwudPWILLxasK4gWzRQ== --osd-uuid 1e9e98fe-77f1-4c08-a2d4-95b4c1a223fe 2024-09-10T17:22:10.179 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:10.177+0000 7f2fa960f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:10.209 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:10.207+0000 7f2fa960f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:10.210 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:10.208+0000 7f2fa960f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:10.210 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:10.209+0000 7f2fa960f640 -1 bdev(0x561c5b1dfc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:22:10.210 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:10.209+0000 7f2fa960f640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-10T17:22:15.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-10T17:22:15.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:22:15.870 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:22:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:22:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:22:16.444 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:22:16.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:22:16.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-10T17:22:16.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:22:16.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:22:16.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:22:16.471 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:16.469+0000 7fc57b7dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:16.493 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:16.492+0000 7fc57b7dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:16.494 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:16.493+0000 7fc57b7dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:16.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:22:16.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:22:16.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:22:16.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:22:16.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:22:16.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:16.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:22:16.891 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:22:16.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:16.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:17.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:17.866 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:17.866+0000 7fc57b7dd640 -1 Falling back to public interface 2024-09-10T17:22:18.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:18.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:18.317 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:22:18.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:22:18.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:18.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:18.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:19.686 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:19.685+0000 7fc57b7dd640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:22:19.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:19.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:19.709 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:22:19.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:22:19.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:19.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:20.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:21.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:21.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:21.332 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:22:21.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:22:21.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:21.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:21.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:22.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:22.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:22.771 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:22:22.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:22:22.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:22.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:24.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:24.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:24.166 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:22:24.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:22:24.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:24.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:24.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:25.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:25.638 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:22:25.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:25.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:22:25.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:25.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:26.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:27.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:27.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:27.055 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:22:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:22:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:27.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:28.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:28.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:28.553 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:22:28.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:22:28.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:28.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:28.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:29.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:29.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:29.980 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:22:29.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:22:29.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:29.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:30.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:31.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:31.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:31.922 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:22:31.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:22:31.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:31.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:32.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:33.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:33.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:33.773 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:22:33.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:22:33.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:33.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:34.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:35.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:35.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:35.692 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:22:35.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:22:35.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:35.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:36.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:37.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:37.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:37.203 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:22:37.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:22:37.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:37.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:37.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:38.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:38.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:38.904 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:22:38.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:22:38.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:38.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:39.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:40.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:40.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:40.807 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:22:40.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:22:40.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:40.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:41.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:42.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:42.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:42.544 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:22:42.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:22:42.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:42.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:22:42.954 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1349527089,v1:127.0.0.1:6803/1349527089] [v2:127.0.0.1:6804/1349527089,v1:127.0.0.1:6805/1349527089] exists,up 1e9e98fe-77f1-4c08-a2d4-95b4c1a223fe 2024-09-10T17:22:42.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:22:42.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:22:42.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:22:42.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-09-10T17:22:42.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-09-10T17:22:43.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-09-10T17:22:43.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:43.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-10T17:22:43.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_profile 2024-09-10T17:22:43.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-09-10T17:22:43.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-09-10T17:22:43.427 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-09-10T17:22:43.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-09-10T17:22:43.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-10T17:22:43.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-10T17:22:43.530 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:22:43.530 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "" 2024-09-10T17:22:43.530 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:22:43.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-10T17:22:43.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-10T17:22:43.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:43.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:43.540 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:43.540 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:43.540 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:43.541 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:43.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-10T17:22:43.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_profile 2024-09-10T17:22:43.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-09-10T17:22:43.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-09-10T17:22:43.625 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-09-10T17:22:43.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-09-10T17:22:43.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-09-10T17:22:43.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-09-10T17:22:43.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-10T17:22:43.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-09-10T17:22:43.626 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-10T17:22:43.627 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:43.627 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:43.627 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:43.627 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:43.627 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:43.627 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:43.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-10T17:22:43.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:22:43.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-09-10T17:22:43.711 INFO:tasks.workunit.client.0.smithi155.stdout:Original client_res for osd.0 = 0.300000 2024-09-10T17:22:43.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-09-10T17:22:43.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-09-10T17:22:43.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-09-10T17:22:43.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-09-10T17:22:43.713 INFO:tasks.workunit.client.0.smithi155.stdout:client_res_new = .400000 2024-09-10T17:22:43.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-09-10T17:22:43.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-09-10T17:22:44.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-10T17:22:44.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-10T17:22:44.575 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-10T17:22:44.576 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-09-10T17:22:44.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-10T17:22:44.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-10T17:22:44.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-09-10T17:22:44.578 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-10T17:22:44.578 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:44.578 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:44.579 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:44.579 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:44.579 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:44.579 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:44.579 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-10T17:22:44.579 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:22:44.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-09-10T17:22:44.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-10T17:22:44.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-09-10T17:22:44.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-10T17:22:44.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-09-10T17:22:44.768 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:22:44.769 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "" 2024-09-10T17:22:44.769 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:22:44.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-10T17:22:44.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-10T17:22:44.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:44.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:44.779 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:44.779 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:44.779 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:44.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:44.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-10T17:22:44.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_profile 2024-09-10T17:22:44.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-09-10T17:22:44.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-09-10T17:22:44.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-10T17:22:44.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-10T17:22:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-10T17:22:45.276 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-10T17:22:45.276 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-09-10T17:22:45.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-10T17:22:45.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-10T17:22:45.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-10T17:22:45.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:22:45.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-09-10T17:22:45.363 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-10T17:22:45.363 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-09-10T17:22:45.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-10T17:22:45.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-09-10T17:22:45.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-09-10T17:22:50.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-10T17:22:51.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-09-10T17:22:51.190 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-09-10T17:22:51.190 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-09-10T17:22:51.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-10T17:22:51.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:22:51.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-09-10T17:22:51.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-09-10T17:22:51.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-09-10T17:22:51.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-10T17:22:51.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-09-10T17:22:51.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:22:51.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:22:51.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:22:51.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:22:51.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:22:51.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:22:51.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:22:51.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:22:51.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:22:51.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:22:51.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:22:51.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:22:51.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:22:51.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:22:51.388 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:22:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:22:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:22:51.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:51.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:22:51.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:22:51.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:22:51.396 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:22:51.396 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:51.396 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:51.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:22:51.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:22:51.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:22:51.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-10T17:22:51.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:22:51.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:22:51.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:22:51.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:22:51.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:22:51.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:22:51.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:22:51.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:22:51.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:22:51.400 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:22:51.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:22:51.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:22:51.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:22:51.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:22:51.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:22:51.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:51.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:22:51.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:22:51.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:51.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:22:51.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:22:51.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:22:51.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:22:51.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:51.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:51.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:22:51.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:22:51.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:22:51.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:22:51.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:22:51.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:22:51.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:22:51.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:22:51.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:22:51.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:22:51.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:22:51.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:22:51.411 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:22:51.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:51.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:22:51.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:22:51.413 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:51.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:22:51.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:22:51.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:22:51.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:22:51.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:51.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:51.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:22:51.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:22:51.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:22:51.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:22:51.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:22:51.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:51.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:51.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:22:51.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:22:51.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:22:51.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:22:51.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:22:51.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:22:51.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:22:51.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:22:51.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:22:51.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:51.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:22:51.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:22:51.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:22:51.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:22:51.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:22:51.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:22:51.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:22:51.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:51.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:51.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:22:51.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:22:51.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:22:51.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:22:51.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:22:51.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:51.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:51.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:22:51.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-10T17:22:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-10T17:22:51.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:22:51.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:22:51.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:22:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:22:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:51.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:22:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:22:52.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:22:52.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:22:52.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:22:52.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:22:52.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:22:52.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:22:52.059 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:22:52.060 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:22:52.060 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:22:52.060 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:52.060 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:52.060 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:52.060 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:22:52.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:22:52.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get fsid 2024-09-10T17:22:52.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:22:52.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:22:52.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:22:52.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:22:52.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:22:52.139 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:22:52.140 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:22:52.140 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:22:52.140 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:22:52.140 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:52.140 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:52.140 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:22:52.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:22:52.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get mon_host 2024-09-10T17:22:52.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-09-10T17:22:52.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-10T17:22:52.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:22:52.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:22:52.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:22:52.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-10T17:22:52.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:22:53.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:22:53.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:22:53.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:22:53.054 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:22:53.054 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:53.054 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:53.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:22:53.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:22:53.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:22:53.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-10T17:22:53.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-10T17:22:53.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-10T17:22:53.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-10T17:22:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-10T17:22:53.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:22:53.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:22:53.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:22:53.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:22:53.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:22:53.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-10T17:22:53.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-10T17:22:53.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:22:53.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6fc26f00-f329-4761-a067-e6a9cff92201 2024-09-10T17:22:53.088 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 6fc26f00-f329-4761-a067-e6a9cff92201 2024-09-10T17:22:53.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6fc26f00-f329-4761-a067-e6a9cff92201' 2024-09-10T17:22:53.088 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:22:53.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBtgOBmne8JBhAA8chOFPLXYxj5Z21+94r/Fg== 2024-09-10T17:22:53.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBtgOBmne8JBhAA8chOFPLXYxj5Z21+94r/Fg=="}' 2024-09-10T17:22:53.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6fc26f00-f329-4761-a067-e6a9cff92201 -i td/mclock-config/0/new.json 2024-09-10T17:22:53.382 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:22:53.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-10T17:22:53.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBtgOBmne8JBhAA8chOFPLXYxj5Z21+94r/Fg== --osd-uuid 6fc26f00-f329-4761-a067-e6a9cff92201 2024-09-10T17:22:53.422 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:53.420+0000 7f4f13e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:53.451 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:53.449+0000 7f4f13e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:53.452 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:53.450+0000 7f4f13e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:53.453 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:53.451+0000 7f4f13e1c640 -1 bdev(0x55a0bcb9dc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:22:53.453 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:53.451+0000 7f4f13e1c640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-10T17:22:58.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-10T17:22:58.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:22:58.796 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:22:58.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:22:58.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:22:59.332 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:22:59.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:22:59.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-10T17:22:59.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:22:59.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:22:59.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:22:59.359 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:59.357+0000 7fd0ef42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:59.360 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:59.359+0000 7fd0ef42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:59.362 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:22:59.360+0000 7fd0ef42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:59.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:22:59.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:22:59.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:22:59.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:22:59.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:22:59.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:59.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:22:59.767 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:22:59.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:59.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:00.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:00.197 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:00.196+0000 7fd0ef42e640 -1 Falling back to public interface 2024-09-10T17:23:01.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:01.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:01.185 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:23:01.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:23:01.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:01.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:01.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:01.945 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:01.944+0000 7fd0ef42e640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:23:02.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:02.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:02.577 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:23:02.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:23:02.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:02.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:03.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:04.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:04.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:04.042 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:23:04.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:23:04.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:04.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:04.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:05.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:05.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:05.471 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:23:05.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:23:05.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:05.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:05.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:06.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:06.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:06.864 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:23:06.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:23:06.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:06.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:07.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:08.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:08.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:08.282 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:23:08.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:23:08.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:08.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:08.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:09.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:09.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:09.722 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:23:09.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:23:09.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:09.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:10.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:11.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:11.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:11.133 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:23:11.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:23:11.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:11.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:11.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:12.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:12.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:12.536 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:23:12.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:23:12.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:12.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:13.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:14.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:14.078 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:23:14.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:14.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:23:14.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:14.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:14.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:15.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:15.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:15.647 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:23:15.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:23:15.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:15.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:16.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:17.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:17.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:17.503 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:23:17.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:23:17.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:17.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:18.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:19.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:19.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:19.002 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:23:19.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:23:19.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:19.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:19.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:20.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:20.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:20.503 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:23:20.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:23:20.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:20.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:21.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:22.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:22.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:22.248 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:23:22.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:23:22.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:22.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:22.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:23.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:23.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:23.975 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:23:23.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:23:23.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:23.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:24.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:25.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:25.673 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:23:25.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:25.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:23:25.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:25.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:26.058 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3152004142,v1:127.0.0.1:6803/3152004142] [v2:127.0.0.1:6804/3152004142,v1:127.0.0.1:6805/3152004142] exists,up 6fc26f00-f329-4761-a067-e6a9cff92201 2024-09-10T17:23:26.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:23:26.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:23:26.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:23:26.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-09-10T17:23:26.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-09-10T17:23:26.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-09-10T17:23:26.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-10T17:23:26.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_profile 2024-09-10T17:23:26.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-10T17:23:26.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-09-10T17:23:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-09-10T17:23:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-09-10T17:23:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-09-10T17:23:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:26.529 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:26.530 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:23:26.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-10T17:23:26.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-10T17:23:26.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-09-10T17:23:26.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-09-10T17:23:26.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:26.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-09-10T17:23:26.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-09-10T17:23:27.168 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-10T17:23:27.169 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-10T17:23:27.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-10T17:23:27.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-10T17:23:27.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-10T17:23:29.191 INFO:tasks.workunit.client.0.smithi155.stdout:Check configuration values - Attempt#: 0 2024-09-10T17:23:29.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-10T17:23:29.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-10T17:23:29.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-10T17:23:29.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-10T17:23:29.622 INFO:tasks.workunit.client.0.smithi155.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-10T17:23:29.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-09-10T17:23:29.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:29.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-10T17:23:29.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-10T17:23:29.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-09-10T17:23:29.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-10T17:23:29.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-10T17:23:30.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-09-10T17:23:30.003 INFO:tasks.workunit.client.0.smithi155.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-10T17:23:30.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-10T17:23:30.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-09-10T17:23:30.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:30.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-09-10T17:23:30.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:30.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-10T17:23:30.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-10T17:23:30.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-10T17:23:30.006 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-10T17:23:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:23:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:23:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:23:30.007 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-10T17:23:30.007 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-10T17:23:30.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-09-10T17:23:30.092 INFO:tasks.workunit.client.0.smithi155.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-10T17:23:30.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-10T17:23:30.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-09-10T17:23:30.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:30.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-09-10T17:23:30.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:30.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-10T17:23:30.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-10T17:23:30.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-10T17:23:30.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-10T17:23:30.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-10T17:23:30.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-10T17:23:30.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:23:30.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-09-10T17:23:30.181 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-09-10T17:23:30.181 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:30.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-09-10T17:23:30.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-10T17:23:30.756 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-10T17:23:30.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-10T17:23:30.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-10T17:23:30.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-10T17:23:30.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-10T17:23:32.759 INFO:tasks.workunit.client.0.smithi155.stdout:Check configuration values - Attempt#: 0 2024-09-10T17:23:32.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-10T17:23:32.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-10T17:23:33.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-10T17:23:33.165 INFO:tasks.workunit.client.0.smithi155.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-10T17:23:33.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-10T17:23:33.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-09-10T17:23:33.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:33.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-10T17:23:33.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-10T17:23:33.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-09-10T17:23:33.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-10T17:23:33.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-10T17:23:33.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-09-10T17:23:33.546 INFO:tasks.workunit.client.0.smithi155.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-10T17:23:33.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-10T17:23:33.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-09-10T17:23:33.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:33.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-09-10T17:23:33.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:33.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-10T17:23:33.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-10T17:23:33.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-10T17:23:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-10T17:23:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:23:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:23:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:33.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:23:33.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-10T17:23:33.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:23:33.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-09-10T17:23:33.636 INFO:tasks.workunit.client.0.smithi155.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-10T17:23:33.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-10T17:23:33.636 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-09-10T17:23:33.637 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:33.637 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-09-10T17:23:33.637 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-10T17:23:33.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-10T17:23:33.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-10T17:23:33.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-10T17:23:33.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-09-10T17:23:33.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:23:33.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:23:33.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:23:33.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:23:33.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:23:33.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:23:33.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:23:33.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:23:33.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:23:33.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:23:33.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:23:33.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:23:33.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:23:33.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:23:33.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:23:33.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:33.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:23:33.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:23:33.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:33.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:23:33.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:23:33.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:23:33.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:23:33.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:33.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:33.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:23:33.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:23:33.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:23:33.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:23:33.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:23:33.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:23:33.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:23:33.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:23:33.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:23:33.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:23:33.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:23:33.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:33.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:23:33.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:23:33.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:33.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:23:33.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:23:33.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:23:33.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:23:33.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:33.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:33.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:23:33.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:23:33.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:23:33.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:23:33.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:23:33.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:23:33.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:23:33.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:23:33.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:23:33.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:23:33.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:23:33.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:33.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:23:33.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:23:33.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:33.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:23:33.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:23:33.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:23:33.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:23:33.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:33.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:33.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:23:33.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:23:33.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:23:33.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:23:33.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:23:33.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:33.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:33.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:23:33.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:23:33.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:23:33.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:23:33.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:23:33.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:23:33.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:23:33.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:23:33.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:23:33.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:23:33.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:23:33.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:23:33.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:23:33.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:23:33.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:33.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:23:33.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:23:33.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:33.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:23:33.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:23:33.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:23:33.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:23:33.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:33.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:33.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:23:33.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:23:33.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:23:33.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:23:33.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:23:33.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:33.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:33.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:23:33.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:23:33.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:23:33.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:23:33.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:23:33.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-09-10T17:23:33.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-10T17:23:33.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:23:33.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:23:33.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:23:33.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-10T17:23:33.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-10T17:23:34.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:23:34.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:23:34.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:23:34.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:23:34.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:34.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:34.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:23:34.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:23:34.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:23:34.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:23:34.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:23:34.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:23:34.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:23:34.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:23:34.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:23:34.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:23:34.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:23:34.458 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:34.458 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:34.458 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:34.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:23:34.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:23:34.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get fsid 2024-09-10T17:23:34.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:23:34.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:23:34.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:23:34.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:23:34.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:23:34.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:23:34.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:23:34.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:23:34.541 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:34.541 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:34.541 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:34.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:23:34.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:23:34.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get mon_host 2024-09-10T17:23:34.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-09-10T17:23:34.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-10T17:23:34.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:23:34.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:23:34.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:23:34.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-10T17:23:34.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:23:35.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:23:35.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:23:35.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:23:35.461 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:23:35.462 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:35.462 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:35.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:23:35.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:23:35.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:23:35.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-10T17:23:35.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-10T17:23:35.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:23:35.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:23:35.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:23:35.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-10T17:23:35.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:23:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:23:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:23:35.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-10T17:23:35.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-10T17:23:35.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:23:35.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=096d2724-c93a-4edc-9ab4-ff7ef04e1282 2024-09-10T17:23:35.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 096d2724-c93a-4edc-9ab4-ff7ef04e1282' 2024-09-10T17:23:35.495 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 096d2724-c93a-4edc-9ab4-ff7ef04e1282 2024-09-10T17:23:35.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:23:35.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCXgOBmN9dJHhAA48Rd1mHEErqZI/DcZUh0CA== 2024-09-10T17:23:35.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCXgOBmN9dJHhAA48Rd1mHEErqZI/DcZUh0CA=="}' 2024-09-10T17:23:35.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 096d2724-c93a-4edc-9ab4-ff7ef04e1282 -i td/mclock-config/0/new.json 2024-09-10T17:23:35.780 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:23:35.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-10T17:23:35.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCXgOBmN9dJHhAA48Rd1mHEErqZI/DcZUh0CA== --osd-uuid 096d2724-c93a-4edc-9ab4-ff7ef04e1282 2024-09-10T17:23:35.820 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:35.818+0000 7fc79b01e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:35.849 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:35.847+0000 7fc79b01e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:35.850 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:35.848+0000 7fc79b01e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:35.850 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:35.849+0000 7fc79b01e640 -1 bdev(0x5638e5479c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:23:35.850 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:35.849+0000 7fc79b01e640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-10T17:23:41.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-10T17:23:41.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:23:41.559 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:23:41.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:23:41.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:23:42.100 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:23:42.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:23:42.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-10T17:23:42.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:23:42.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:23:42.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:23:42.127 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:42.125+0000 7f28b8320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:42.143 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:42.142+0000 7f28b8320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:42.144 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:42.143+0000 7f28b8320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:42.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:23:42.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:23:42.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:23:42.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:23:42.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:23:42.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:42.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:23:42.557 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:23:42.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:42.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:42.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:43.493 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:43.492+0000 7f28b8320640 -1 Falling back to public interface 2024-09-10T17:23:43.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:43.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:43.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:23:43.939 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:23:43.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:43.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:44.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:45.087 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:23:45.085+0000 7f28b8320640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:23:45.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:45.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:45.369 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:23:45.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:23:45.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:45.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:45.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:46.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:46.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:46.844 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:23:46.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:23:46.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:46.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:47.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:48.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:48.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:48.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:23:48.252 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:23:48.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:48.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:49.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:49.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:49.682 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:23:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:23:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:50.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:51.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:51.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:51.205 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:23:51.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:23:51.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:51.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:51.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:52.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:52.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:52.622 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:23:52.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:23:52.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:52.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:53.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:54.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:54.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:54.018 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:23:54.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:23:54.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:54.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:54.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:55.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:55.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:55.410 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:23:55.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:23:55.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:55.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:55.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:56.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:56.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:56.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:23:56.887 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:23:56.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:56.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:57.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:58.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:58.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:58.605 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:23:58.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:23:58.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:58.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:59.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:00.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:00.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:00.107 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:24:00.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:24:00.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:00.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:00.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:01.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:01.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:01.743 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:24:01.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:24:01.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:01.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:02.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:03.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:03.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:03.790 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:24:03.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:24:03.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:03.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:04.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:05.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:05.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:05.364 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:24:05.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:24:05.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:05.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:05.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:06.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:06.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:06.821 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:24:06.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:24:06.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:06.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:07.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:08.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:08.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:08.473 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:24:08.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:24:08.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:08.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:08.889 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/635241269,v1:127.0.0.1:6803/635241269] [v2:127.0.0.1:6804/635241269,v1:127.0.0.1:6805/635241269] exists,up 096d2724-c93a-4edc-9ab4-ff7ef04e1282 2024-09-10T17:24:08.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:24:08.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:24:08.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:24:08.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-09-10T17:24:09.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-09-10T17:24:09.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-09-10T17:24:09.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-09-10T17:24:09.309 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-10T17:24:09.309 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:24:09.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:24:09.310 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:09.310 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:09.310 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:09.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:24:09.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-10T17:24:09.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_profile 2024-09-10T17:24:09.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-10T17:24:09.396 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-09-10T17:24:09.396 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-09-10T17:24:09.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:09.397 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:09.398 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:09.398 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:24:09.398 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-10T17:24:09.398 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-10T17:24:09.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-09-10T17:24:09.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-09-10T17:24:09.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:09.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-09-10T17:24:09.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-09-10T17:24:09.603 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:24:09.603 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-09-10T17:24:09.603 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:24:09.612 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-10T17:24:09.614 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-10T17:24:09.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-10T17:24:09.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-10T17:24:09.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-10T17:24:11.616 INFO:tasks.workunit.client.0.smithi155.stdout:Check configuration values - Attempt#: 0 2024-09-10T17:24:11.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-10T17:24:11.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-10T17:24:12.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-10T17:24:12.052 INFO:tasks.workunit.client.0.smithi155.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-10T17:24:12.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-10T17:24:12.053 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-09-10T17:24:12.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:12.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-10T17:24:12.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-10T17:24:12.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-09-10T17:24:12.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-10T17:24:12.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-10T17:24:12.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-09-10T17:24:12.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-10T17:24:12.436 INFO:tasks.workunit.client.0.smithi155.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-10T17:24:12.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-09-10T17:24:12.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:12.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-09-10T17:24:12.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:12.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-10T17:24:12.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-10T17:24:12.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-10T17:24:12.439 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-10T17:24:12.440 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:24:12.440 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:24:12.440 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:12.440 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:12.440 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:12.440 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:24:12.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-10T17:24:12.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-10T17:24:12.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-09-10T17:24:12.523 INFO:tasks.workunit.client.0.smithi155.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-10T17:24:12.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-10T17:24:12.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-09-10T17:24:12.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:12.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-09-10T17:24:12.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:12.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-10T17:24:12.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-10T17:24:12.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-10T17:24:12.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:24:12.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-10T17:24:12.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:24:12.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-09-10T17:24:12.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-09-10T17:24:12.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:12.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-09-10T17:24:12.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-09-10T17:24:12.716 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:24:12.716 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-09-10T17:24:12.716 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:24:12.725 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-10T17:24:12.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-10T17:24:12.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-10T17:24:12.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-10T17:24:12.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-10T17:24:14.729 INFO:tasks.workunit.client.0.smithi155.stdout:Check configuration values - Attempt#: 0 2024-09-10T17:24:14.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-10T17:24:14.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-10T17:24:15.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-10T17:24:15.163 INFO:tasks.workunit.client.0.smithi155.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-10T17:24:15.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-10T17:24:15.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-09-10T17:24:15.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:15.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-10T17:24:15.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-10T17:24:15.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-09-10T17:24:15.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-10T17:24:15.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-10T17:24:15.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-09-10T17:24:15.545 INFO:tasks.workunit.client.0.smithi155.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-10T17:24:15.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-10T17:24:15.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-09-10T17:24:15.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:15.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-09-10T17:24:15.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:15.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-10T17:24:15.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-10T17:24:15.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-10T17:24:15.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-10T17:24:15.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:24:15.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:24:15.550 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:15.550 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:15.550 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:15.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:24:15.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-10T17:24:15.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-10T17:24:15.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-09-10T17:24:15.633 INFO:tasks.workunit.client.0.smithi155.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-10T17:24:15.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-10T17:24:15.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-09-10T17:24:15.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:15.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-09-10T17:24:15.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:24:15.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:24:15.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:24:15.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:24:15.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:24:15.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:24:15.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:24:15.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:24:15.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:24:15.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:24:15.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:24:15.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:15.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:24:15.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:24:15.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:15.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:24:15.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:24:15.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:24:15.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:24:15.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:15.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:15.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:24:15.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:24:15.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:24:15.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-10T17:24:15.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:24:15.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:24:15.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:24:15.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:24:15.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:24:15.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:24:15.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:24:15.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:24:15.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:24:15.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:24:15.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:24:15.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:24:15.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:24:15.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:24:15.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:24:15.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:15.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:24:15.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:24:15.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:15.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:24:15.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:24:15.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:24:15.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:24:15.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:15.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:15.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:24:15.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:24:15.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:24:15.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:24:15.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:15.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:24:15.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:24:15.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:24:15.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:24:15.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:15.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:15.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:24:15.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:24:15.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:24:15.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:24:15.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:24:15.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:15.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:15.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:24:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:24:15.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:24:15.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:24:15.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:24:15.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:24:15.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:24:15.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:24:15.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:24:15.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:24:15.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:24:15.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:24:15.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:24:15.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:24:15.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:24:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:24:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:24:15.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:15.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:24:15.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:24:15.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:24:15.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:24:15.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:15.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:15.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:24:15.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:24:15.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:24:15.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-10T17:24:15.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:24:15.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:15.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:15.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27164 2024-09-10T17:24:15.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:24:15.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:24:15.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:24:15.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-10T17:24:15.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-10T17:24:15.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-10T17:24:15.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:24:15.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:24:15.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:24:15.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-10T17:24:15.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-10T17:24:16.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:24:16.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:24:16.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:24:16.258 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:24:16.258 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:16.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:24:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:24:16.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:24:16.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:24:16.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:24:16.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:24:16.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:24:16.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:24:16.452 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:24:16.452 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:24:16.452 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:24:16.453 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:16.453 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:16.453 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:16.453 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:24:16.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:24:16.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get fsid 2024-09-10T17:24:16.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:24:16.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:24:16.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:24:16.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:24:16.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:24:16.534 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:24:16.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:24:16.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:24:16.535 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:16.535 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:16.535 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:16.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-mon.a.asok 2024-09-10T17:24:16.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:24:16.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27164/ceph-mon.a.asok config get mon_host 2024-09-10T17:24:16.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-10T17:24:16.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-10T17:24:16.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:24:16.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:24:16.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:24:16.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-10T17:24:16.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:24:17.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:24:17.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:24:17.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:24:17.456 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:24:17.456 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:17.456 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:17.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:24:17.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:24:17.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:24:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:24:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-10T17:24:17.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:24:17.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7fa4003f-77a6-451f-8795-81879699107a 2024-09-10T17:24:17.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7fa4003f-77a6-451f-8795-81879699107a' 2024-09-10T17:24:17.490 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 7fa4003f-77a6-451f-8795-81879699107a 2024-09-10T17:24:17.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:24:17.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDBgOBmT7j4HRAAPmCBeE0X9zJwhzlI/1ETuw== 2024-09-10T17:24:17.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDBgOBmT7j4HRAAPmCBeE0X9zJwhzlI/1ETuw=="}' 2024-09-10T17:24:17.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7fa4003f-77a6-451f-8795-81879699107a -i td/mclock-config/0/new.json 2024-09-10T17:24:17.783 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:24:17.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-10T17:24:17.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDBgOBmT7j4HRAAPmCBeE0X9zJwhzlI/1ETuw== --osd-uuid 7fa4003f-77a6-451f-8795-81879699107a 2024-09-10T17:24:17.823 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:17.821+0000 7f7f9302c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:17.853 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:17.851+0000 7f7f9302c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:17.853 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:17.852+0000 7f7f9302c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:17.854 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:17.853+0000 7f7f9302c640 -1 bdev(0x555982197c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:24:17.854 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:17.853+0000 7f7f9302c640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-10T17:24:23.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-10T17:24:23.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:24:23.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:24:23.354 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:24:23.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:24:23.915 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:24:23.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:24:23.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=951add96-468a-42b4-ae9d-a7cb2a5eff17 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27164/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-10T17:24:23.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:24:23.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:24:23.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:24:23.942 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:23.941+0000 7ffa5be1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:23.944 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:23.943+0000 7ffa5be1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:23.945 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:23.944+0000 7ffa5be1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:24:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:24:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:24:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:24:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:24:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:24:24.299 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:24:24.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:24.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:24.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:25.299 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:25.298+0000 7ffa5be1e640 -1 Falling back to public interface 2024-09-10T17:24:25.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:25.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:25.712 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:24:25.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:24:25.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:25.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:26.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:27.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:27.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:27.130 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:24:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:24:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:27.131 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:27.130+0000 7ffa5be1e640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:24:27.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:28.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:28.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:28.566 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:24:28.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:24:28.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:28.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:28.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:29.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:29.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:29.981 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:24:29.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:24:29.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:29.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:30.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:31.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:31.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:31.392 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:24:31.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:24:31.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:31.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:31.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:32.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:32.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:32.858 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:24:32.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:24:32.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:32.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:33.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:34.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:34.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:34.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:24:34.274 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:24:34.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:34.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:34.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:35.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:35.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:35.700 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:24:35.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:24:35.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:35.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:36.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:37.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:37.180 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:24:37.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:37.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:24:37.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:37.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:37.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:38.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:38.591 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:24:38.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:38.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:24:38.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:38.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:39.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:40.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:40.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:40.529 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:24:40.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:24:40.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:40.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:41.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:42.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:42.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:42.105 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:24:42.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:24:42.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:42.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:42.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:43.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:43.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:43.865 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:24:43.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:24:43.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:43.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:44.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:45.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:45.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:45.653 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:24:45.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:24:45.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:45.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:46.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:47.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:47.052 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:24:47.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:47.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:24:47.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:47.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:48.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:48.749 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:24:48.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:48.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:24:48.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:48.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:49.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:50.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:50.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:50.468 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:24:50.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:24:50.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:50.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:50.897 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/605650218,v1:127.0.0.1:6803/605650218] [v2:127.0.0.1:6804/605650218,v1:127.0.0.1:6805/605650218] exists,up 7fa4003f-77a6-451f-8795-81879699107a 2024-09-10T17:24:50.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:24:50.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:24:50.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:24:50.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-10T17:24:50.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-09-10T17:24:50.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-09-10T17:24:50.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-09-10T17:24:50.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-10T17:24:51.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-10T17:24:53.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:24:53.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:24:53.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:24:53.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:53.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:53.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:53.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:24:53.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:24:53.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-10T17:24:53.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-09-10T17:24:53.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-09-10T17:24:53.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-10T17:24:54.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-10T17:24:54.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-10T17:24:56.561 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-10T17:24:56.561 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:24:56.561 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:24:56.562 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:56.562 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:56.562 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:56.562 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27164/ceph-osd.0.asok 2024-09-10T17:24:56.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-10T17:24:56.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27164/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-10T17:24:56.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-09-10T17:24:56.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:24:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:24:56.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:24:56.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:24:56.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:24:56.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:24:56.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:24:56.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:24:56.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:24:56.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:56.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:24:56.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:24:56.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:56.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:24:56.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:24:56.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:24:56.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:24:56.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:56.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:56.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:24:56.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:24:56.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:24:56.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:24:56.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:24:56.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:24:56.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:24:56.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:24:56.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:24:56.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:24:56.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:24:56.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:56.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:24:56.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:24:56.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:56.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:24:56.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:24:56.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:24:56.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:24:56.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:56.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:56.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:24:56.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:24:56.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:24:56.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T17:24:56.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/mclock-config 0 2024-09-10T17:24:56.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-10T17:24:56.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T17:24:56.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-10T17:24:56.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:24:56.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:24:56.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:24:56.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:24:56.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:24:56.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:24:56.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:24:56.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:24:56.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:24:56.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:24:56.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:24:56.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:24:56.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:56.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:24:56.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:24:56.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:56.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:24:56.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T17:24:56.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-10T17:24:56.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:24:56.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:56.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27164 2024-09-10T17:24:56.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27164 2024-09-10T17:24:56.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:24:56.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:24:56.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T17:24:56.798 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T17:24:56.798 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T17:24:56.824 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-09-10T17:24:56.824 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-09-10T17:24:56.886 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T17:24:56.892 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T17:24:56.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:24:56.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:24:56.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/network-ping 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-09-10T17:24:56.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none ' 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-09-10T17:24:56.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T17:24:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-09-10T17:24:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_network_ping_test2' 2024-09-10T17:24:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-10T17:24:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-10T17:24:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2024-09-10T17:24:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:24:56.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:24:56.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:24:56.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:24:56.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:24:56.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:24:56.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:24:56.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:56.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:24:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.41145 2024-09-10T17:24:56.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:24:56.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:24:56.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2024-09-10T17:24:56.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:24:56.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:56.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:24:56.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.41145 2024-09-10T17:24:56.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:24:56.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T17:24:56.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T17:24:56.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:24:56.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-10T17:24:56.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-09-10T17:24:56.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-09-10T17:24:56.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-09-10T17:24:56.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2024-09-10T17:24:56.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:24:56.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:24:56.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:24:56.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2024-09-10T17:24:56.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-10T17:24:57.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:24:57.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:24:57.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:24:57.389 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:24:57.389 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:57.389 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:24:57.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:24:57.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:24:57.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:24:57.600 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:57.601 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:57.601 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:24:57.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mon.a.asok 2024-09-10T17:24:57.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:24:57.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.41145/ceph-mon.a.asok config get fsid 2024-09-10T17:24:57.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:24:57.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:24:57.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:24:57.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:24:57.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mon.a.asok 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:24:57.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.41145/ceph-mon.a.asok config get mon_host 2024-09-10T17:24:57.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-09-10T17:24:57.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2024-09-10T17:24:57.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:24:57.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:24:57.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:24:57.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2024-09-10T17:24:57.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:24:58.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:24:58.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:24:58.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:24:58.611 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:24:58.611 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:58.611 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:24:58.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:24:58.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:24:58.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:24:58.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-09-10T17:24:58.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-10T17:24:58.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:24:58.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:24:58.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:24:58.643 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:24:58.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:24:58.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:24:58.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:24:58.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2024-09-10T17:24:58.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:24:58.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ee9c09de-c118-406b-bd02-53a05a945cda 2024-09-10T17:24:58.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ee9c09de-c118-406b-bd02-53a05a945cda' 2024-09-10T17:24:58.646 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 ee9c09de-c118-406b-bd02-53a05a945cda 2024-09-10T17:24:58.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:24:58.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDqgOBmHFNkJxAAoVIdBT5V/su0ejFz9/pwOQ== 2024-09-10T17:24:58.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDqgOBmHFNkJxAAoVIdBT5V/su0ejFz9/pwOQ=="}' 2024-09-10T17:24:58.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ee9c09de-c118-406b-bd02-53a05a945cda -i td/network-ping/0/new.json 2024-09-10T17:24:58.931 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:24:58.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2024-09-10T17:24:58.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqgOBmHFNkJxAAoVIdBT5V/su0ejFz9/pwOQ== --osd-uuid ee9c09de-c118-406b-bd02-53a05a945cda 2024-09-10T17:24:58.972 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:58.970+0000 7fbd3522e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:58.999 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:58.998+0000 7fbd3522e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:59.000 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:58.999+0000 7fbd3522e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:59.001 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:58.999+0000 7fbd3522e640 -1 bdev(0x556717561c00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:24:59.001 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:24:58.999+0000 7fbd3522e640 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-10T17:25:04.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-10T17:25:04.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:25:04.690 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:25:04.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:25:04.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:25:05.269 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:25:05.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:25:05.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:25:05.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:25:05.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:25:05.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:25:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:05.296+0000 7f4f48819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:05.304 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:05.302+0000 7f4f48819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:05.305 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:05.304+0000 7f4f48819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:05.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:25:05.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:25:05.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:25:05.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:25:05.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:25:05.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:05.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:25:05.726 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:25:05.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:05.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:06.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:06.663 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:06.662+0000 7f4f48819640 -1 Falling back to public interface 2024-09-10T17:25:07.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:07.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:07.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:25:07.145 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:25:07.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:07.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:07.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:08.506 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:08.504+0000 7f4f48819640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:25:08.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:08.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:08.579 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:25:08.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:25:08.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:08.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:09.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:10.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:10.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:10.126 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:25:10.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:25:10.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:10.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:10.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:11.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:11.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:11.563 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:25:11.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:25:11.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:11.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:11.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:12.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:12.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:12.977 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:25:12.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:25:12.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:12.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:13.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:14.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:14.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:14.434 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:25:14.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:25:14.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:14.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:14.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:15.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:15.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:15.834 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:25:15.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:25:15.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:15.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:16.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:17.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:17.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:17.295 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:25:17.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:25:17.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:17.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:17.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:18.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:18.716 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:25:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:25:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:19.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:20.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:20.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:20.111 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:25:20.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:25:20.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:20.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:20.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:21.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:21.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:21.800 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:25:21.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:25:21.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:21.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:22.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:23.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:23.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:23.358 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:25:23.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:25:23.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:23.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:24.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:25.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:25.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:25.116 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:25:25.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:25:25.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:25.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:25.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:26.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:26.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:26.655 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:25:26.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:25:26.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:26.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:27.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:28.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:28.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:28.291 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:25:28.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:25:28.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:28.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:29.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:30.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:30.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:30.101 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:25:30.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:25:30.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:30.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:30.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:31.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:31.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:31.559 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:25:31.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:25:31.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:31.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:25:32.018 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4045004289,v1:127.0.0.1:6803/4045004289] [v2:127.0.0.1:6804/4045004289,v1:127.0.0.1:6805/4045004289] exists,up ee9c09de-c118-406b-bd02-53a05a945cda 2024-09-10T17:25:32.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:25:32.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:25:32.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:25:32.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-09-10T17:25:32.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-10T17:25:32.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:25:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-10T17:25:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=52a988bc-3261-4350-ab0e-eb2eaa10613d 2024-09-10T17:25:32.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 52a988bc-3261-4350-ab0e-eb2eaa10613d' 2024-09-10T17:25:32.026 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 52a988bc-3261-4350-ab0e-eb2eaa10613d 2024-09-10T17:25:32.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:25:32.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAMgeBmncNXAhAAotY1gUP5K5Ka+wJZGfsIIA== 2024-09-10T17:25:32.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAMgeBmncNXAhAAotY1gUP5K5Ka+wJZGfsIIA=="}' 2024-09-10T17:25:32.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 52a988bc-3261-4350-ab0e-eb2eaa10613d -i td/network-ping/1/new.json 2024-09-10T17:25:32.567 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:25:32.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2024-09-10T17:25:32.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMgeBmncNXAhAAotY1gUP5K5Ka+wJZGfsIIA== --osd-uuid 52a988bc-3261-4350-ab0e-eb2eaa10613d 2024-09-10T17:25:32.610 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:32.608+0000 7f38ea442640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:32.611 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:32.610+0000 7f38ea442640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:32.612 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:32.611+0000 7f38ea442640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:32.613 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:32.611+0000 7f38ea442640 -1 bdev(0x560b4fa3dc00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:25:32.613 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:32.611+0000 7f38ea442640 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-10T17:25:38.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-10T17:25:38.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:25:38.764 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-10T17:25:38.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:25:38.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:25:39.303 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-10T17:25:39.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:25:39.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:25:39.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:25:39.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:25:39.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:25:39.332 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:39.331+0000 7f38c8814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:39.362 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:39.362+0000 7f38c8814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:39.363 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:39.363+0000 7f38c8814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:39.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:25:39.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:25:39.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:25:39.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:25:39.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:25:39.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:39.727 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:25:39.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:25:39.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:39.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:40.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:40.960 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:40.959+0000 7f38c8814640 -1 Falling back to public interface 2024-09-10T17:25:41.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:41.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:41.153 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:25:41.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:25:41.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:41.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:41.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:42.541 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:25:42.539+0000 7f38c8814640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:25:42.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:42.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:42.588 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:25:42.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:25:42.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:42.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:43.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:44.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:44.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:44.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:25:44.203 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:25:44.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:44.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:45.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:45.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:45.668 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:25:45.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:25:45.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:45.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:46.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:47.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:47.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:47.086 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:25:47.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:25:47.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:47.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:48.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:48.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:48.551 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:25:48.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:25:48.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:48.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:49.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:50.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:50.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:50.053 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:25:50.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:25:50.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:50.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:50.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:51.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:51.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:51.487 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:25:51.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:25:51.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:51.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:51.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:52.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:52.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:52.885 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:25:52.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:25:52.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:52.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:53.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:54.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:54.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:54.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:25:54.442 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:25:54.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:54.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:55.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:56.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:56.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:56.112 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:25:56.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:25:56.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:56.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:57.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:58.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:58.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:58.259 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:25:58.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:25:58.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:58.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:25:58.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:59.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:59.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:59.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:25:59.993 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:25:59.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:59.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:26:00.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:01.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:01.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:01.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:26:01.771 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:26:01.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:01.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:26:02.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:03.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:03.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:03.550 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:26:03.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:26:03.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:03.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:26:04.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:05.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:05.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:05.458 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:26:05.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:26:05.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:05.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:26:05.872 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2636485837,v1:127.0.0.1:6811/2636485837] [v2:127.0.0.1:6812/2636485837,v1:127.0.0.1:6813/2636485837] exists,up 52a988bc-3261-4350-ab0e-eb2eaa10613d 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2024-09-10T17:26:05.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:26:05.877 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:26:05.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2024-09-10T17:26:05.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:26:05.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3e9c01ee-f15d-4f82-94b1-c2f00ea68240 2024-09-10T17:26:05.879 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 3e9c01ee-f15d-4f82-94b1-c2f00ea68240 2024-09-10T17:26:05.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 3e9c01ee-f15d-4f82-94b1-c2f00ea68240' 2024-09-10T17:26:05.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:26:05.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAtgeBm/EpiNRAAZ58gBl34AwfYXbA2lcEFaw== 2024-09-10T17:26:05.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAtgeBm/EpiNRAAZ58gBl34AwfYXbA2lcEFaw=="}' 2024-09-10T17:26:05.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3e9c01ee-f15d-4f82-94b1-c2f00ea68240 -i td/network-ping/2/new.json 2024-09-10T17:26:06.380 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:26:06.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2024-09-10T17:26:06.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtgeBm/EpiNRAAZ58gBl34AwfYXbA2lcEFaw== --osd-uuid 3e9c01ee-f15d-4f82-94b1-c2f00ea68240 2024-09-10T17:26:06.424 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:06.423+0000 7f6f73945640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:06.425 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:06.424+0000 7f6f73945640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:06.427 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:06.425+0000 7f6f73945640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:06.427 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:06.426+0000 7f6f73945640 -1 bdev(0x55918044fc00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:26:06.427 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:06.426+0000 7f6f73945640 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-10T17:26:12.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-10T17:26:12.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:26:12.913 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-10T17:26:12.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:26:12.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:26:13.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:26:13.452 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-10T17:26:13.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:26:13.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:26:13.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:26:13.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:26:13.479 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:13.478+0000 7fe37dd58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:13.486 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:13.484+0000 7fe37dd58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:13.487 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:13.486+0000 7fe37dd58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:13.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:26:13.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:26:13.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:26:13.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:26:13.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:26:13.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:13.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:26:13.901 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:26:13.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:13.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:14.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:14.850 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:14.849+0000 7fe37dd58640 -1 Falling back to public interface 2024-09-10T17:26:15.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:15.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:15.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:26:15.327 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:26:15.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:15.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:15.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:16.498 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:26:16.496+0000 7fe37dd58640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:26:16.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:16.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:16.755 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:26:16.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:26:16.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:16.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:17.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:18.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:18.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:18.289 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:26:18.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:26:18.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:18.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:18.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:19.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:19.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:19.721 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:26:19.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:26:19.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:19.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:20.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:21.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:21.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:21.119 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:26:21.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:26:21.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:21.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:21.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:22.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:22.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:22.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:26:22.611 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:26:22.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:22.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:23.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:24.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:24.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:24.052 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:26:24.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:26:24.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:24.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:24.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:25.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:25.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:25.465 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:26:25.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:26:25.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:25.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:25.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:26.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:26.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:26.889 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:26:26.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:26:26.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:26.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:27.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:28.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:28.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:28.437 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:26:28.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:26:28.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:28.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:29.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:30.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:30.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:30.101 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:26:30.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:26:30.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:30.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:30.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:31.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:31.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:31.813 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:26:31.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:26:31.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:31.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:32.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:33.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:33.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:33.565 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:26:33.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:26:33.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:33.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:34.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:35.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:35.335 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:26:35.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:35.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:26:35.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:35.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:36.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:37.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:37.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:37.524 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:26:37.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:26:37.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:37.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:38.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:39.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:39.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:39.625 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:26:39.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:26:39.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:39.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:26:40.040 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2943826712,v1:127.0.0.1:6819/2943826712] [v2:127.0.0.1:6820/2943826712,v1:127.0.0.1:6821/2943826712] exists,up 3e9c01ee-f15d-4f82-94b1-c2f00ea68240 2024-09-10T17:26:40.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:26:40.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:26:40.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:26:40.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-09-10T17:26:45.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-09-10T17:26:45.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-10T17:26:45.531 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'foo' already exists 2024-09-10T17:26:45.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:26:46.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-10T17:26:46.628 INFO:tasks.workunit.client.0.smithi155.stdout:hints = 1 2024-09-10T17:26:46.628 INFO:tasks.workunit.client.0.smithi155.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-10T17:26:46.628 INFO:tasks.workunit.client.0.smithi155.stdout:Object prefix: benchmark_data_smithi155_46597 2024-09-10T17:26:46.628 INFO:tasks.workunit.client.0.smithi155.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T17:26:46.628 INFO:tasks.workunit.client.0.smithi155.stdout: 0 0 0 0 0 0 - 0 2024-09-10T17:26:47.628 INFO:tasks.workunit.client.0.smithi155.stdout: 1 16 53 37 0.144516 0.144531 0.26488 0.365738 2024-09-10T17:26:48.628 INFO:tasks.workunit.client.0.smithi155.stdout: 2 16 109 93 0.181617 0.21875 0.230087 0.315337 2024-09-10T17:26:49.628 INFO:tasks.workunit.client.0.smithi155.stdout: 3 16 173 157 0.204402 0.25 0.235002 0.286807 2024-09-10T17:26:50.628 INFO:tasks.workunit.client.0.smithi155.stdout: 4 16 237 221 0.215794 0.25 0.245936 0.276606 2024-09-10T17:26:51.629 INFO:tasks.workunit.client.0.smithi155.stdout: 5 16 316 300 0.234348 0.308594 0.241176 0.265934 2024-09-10T17:26:52.628 INFO:tasks.workunit.client.0.smithi155.stdout: 6 16 380 364 0.236952 0.25 0.218452 0.259142 2024-09-10T17:26:53.629 INFO:tasks.workunit.client.0.smithi155.stdout: 7 16 445 429 0.23937 0.253906 0.246836 0.256062 2024-09-10T17:26:54.629 INFO:tasks.workunit.client.0.smithi155.stdout: 8 16 509 493 0.240696 0.25 0.259582 0.253024 2024-09-10T17:26:55.629 INFO:tasks.workunit.client.0.smithi155.stdout: 9 16 573 557 0.241725 0.25 0.266553 0.251558 2024-09-10T17:26:56.629 INFO:tasks.workunit.client.0.smithi155.stdout: 10 16 653 637 0.248799 0.3125 0.209868 0.248366 2024-09-10T17:26:57.629 INFO:tasks.workunit.client.0.smithi155.stdout:Total time run: 10.1543 2024-09-10T17:26:57.629 INFO:tasks.workunit.client.0.smithi155.stdout:Total writes made: 653 2024-09-10T17:26:57.629 INFO:tasks.workunit.client.0.smithi155.stdout:Write size: 4096 2024-09-10T17:26:57.629 INFO:tasks.workunit.client.0.smithi155.stdout:Object size: 4096 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Bandwidth (MB/sec): 0.251201 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev Bandwidth: 0.0464773 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Max bandwidth (MB/sec): 0.3125 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Min bandwidth (MB/sec): 0.144531 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Average IOPS: 64 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev IOPS: 11.8982 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Max IOPS: 80 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Min IOPS: 37 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Average Latency(s): 0.247436 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev Latency(s): 0.0583314 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Max latency(s): 0.596705 2024-09-10T17:26:57.630 INFO:tasks.workunit.client.0.smithi155.stdout:Min latency(s): 0.141812 2024-09-10T17:26:57.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-09-10T17:27:07.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-09-10T17:27:07.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:27:07.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:27:08.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:27:08.158 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:27:08.158 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-10T17:27:08.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:27:08.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:08.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:27:08.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-10T17:27:08.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-10T17:27:08.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-10T17:27:08.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:08.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:27:08.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-10T17:27:08.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-10T17:27:08.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2024-09-10T17:27:08.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:08.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:27:08.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-10T17:27:08.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-10T17:27:08.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509452' 2024-09-10T17:27:08.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:08.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-10T17:27:08.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:08.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:27:08.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-10T17:27:08.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:08.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-10T17:27:08.536 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836505 2024-09-10T17:27:08.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-10T17:27:08.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:27:08.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-10T17:27:08.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:08.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-10T17:27:08.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:08.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:27:08.978 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-10T17:27:08.978 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:08.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-10T17:27:08.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-10T17:27:08.979 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672983 2024-09-10T17:27:08.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:27:09.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-10T17:27:09.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:09.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-10T17:27:09.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:09.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:27:09.404 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-10T17:27:09.404 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:09.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-10T17:27:09.405 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509452 2024-09-10T17:27:09.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-10T17:27:09.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:27:09.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-osd.0.asok 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:27:09.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-osd.0.asok dump_osd_network 2024-09-10T17:27:09.908 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:27:09.909 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-10T17:27:09.909 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-10T17:27:09.909 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:27:09.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:09.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:27:09.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:27:09.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:09.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-09-10T17:27:09.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-09-10T17:27:09.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:27:09.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-10T17:27:09.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-10T17:27:09.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-10T17:27:09.983 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:09.983 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:09.983 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:27:09.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mgr.x.asok 2024-09-10T17:27:09.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:27:09.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-mgr.x.asok dump_osd_network 2024-09-10T17:27:09.994 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:27:09.994 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-10T17:27:09.994 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-10T17:27:09.994 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:27:10.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:10.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:27:10.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:27:10.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:10.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-09-10T17:27:10.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-09-10T17:27:10.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:27:10.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-09-10T17:27:10.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:27:10.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:27:10.009 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:10.009 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:10.009 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:27:10.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-osd.0.asok 2024-09-10T17:27:10.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:27:10.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-osd.0.asok dump_osd_network 0 2024-09-10T17:27:10.087 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:27:10.087 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-10T17:27:10.087 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-10T17:27:10.087 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.087 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:06 2024", 2024-09-10T17:27:10.087 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.087 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:10.087 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.378, 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.336, 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.312 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.469, 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.479, 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.479 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.469, 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.479, 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.479 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.271 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:05 2024", 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:10.089 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.370, 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.317, 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.312 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.449, 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.449, 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.449 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.449, 2024-09-10T17:27:10.090 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.449, 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.449 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.212 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:05 2024", 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.346, 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.299, 2024-09-10T17:27:10.091 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.294 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.419, 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.519, 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.519 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.419, 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.519, 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.519 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.167 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.092 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:06 2024", 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.331, 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.317, 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.303 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.398, 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.493, 2024-09-10T17:27:10.093 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.493 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.398, 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.493, 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.493 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.232 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-10T17:27:10.094 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:27:10.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:10.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:27:10.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-09-10T17:27:10.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:10.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-09-10T17:27:10.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:27:10.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:27:10.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-10T17:27:10.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-10T17:27:10.102 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-10T17:27:10.102 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:10.102 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:10.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:27:10.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mgr.x.asok 2024-09-10T17:27:10.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:27:10.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-mgr.x.asok dump_osd_network 0 2024-09-10T17:27:10.173 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:27:10.173 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-10T17:27:10.173 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-10T17:27:10.173 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.173 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:06 2024", 2024-09-10T17:27:10.173 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.173 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:10.173 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.378, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.336, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.312 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.297, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.222, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.222 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.469, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.479, 2024-09-10T17:27:10.174 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.479 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.443 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:05 2024", 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.370, 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.317, 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.312 2024-09-10T17:27:10.175 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.251, 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.216, 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.212 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.449, 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.449, 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.449 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.419 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.176 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:04 2024", 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.298, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.341, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.365 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.240, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.240, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.240 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.177 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.389, 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.503, 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.503 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.235 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:04 2024", 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:10.178 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.302, 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.327, 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.356 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.250, 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.229, 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.229 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.354, 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.498, 2024-09-10T17:27:10.179 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.498 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.193 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:05 2024", 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.180 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.346, 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.320, 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.277 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.252, 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.213, 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.149 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.525, 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.525, 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.525 2024-09-10T17:27:10.181 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.257 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:05 2024", 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.346, 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.299, 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.294 2024-09-10T17:27:10.182 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.233, 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.164, 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.164 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.419, 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.519, 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.519 2024-09-10T17:27:10.183 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.403 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:05 2024", 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.342, 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.335, 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.288 2024-09-10T17:27:10.184 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.249, 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.208, 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.208 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.485, 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.485, 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.485 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.322 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.185 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:05 2024", 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.334, 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.313, 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.271 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.236, 2024-09-10T17:27:10.186 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.187, 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.187 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.500, 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.500, 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.500 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.302 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:04 2024", 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:27:10.187 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.277, 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.309, 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.333 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.223, 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.206, 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.206 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.370, 2024-09-10T17:27:10.188 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.476, 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.476 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.222 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:06 2024", 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.331, 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.317, 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.303 2024-09-10T17:27:10.189 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.272, 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.208, 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.208 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.398, 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.493, 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.493 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.190 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.386 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:04 2024", 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.287, 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.308, 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.316 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.191 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.207, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.186, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.186 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.338, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.445, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.445 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.208 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:05 2024", 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:27:10.192 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.297, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.295, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.260 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.215, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.192, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.170 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.467, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.467, 2024-09-10T17:27:10.193 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.467 2024-09-10T17:27:10.194 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:10.194 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.280 2024-09-10T17:27:10.194 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-10T17:27:10.194 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-10T17:27:10.194 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:27:10.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:10.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:27:10.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-09-10T17:27:10.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:10.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-09-10T17:27:10.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:27:10.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-09-10T17:27:30.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-09-10T17:27:30.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:27:30.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:27:30.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:27:30.645 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:27:30.645 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-10T17:27:30.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:27:30.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:30.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:27:30.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836511 2024-09-10T17:27:30.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836511 2024-09-10T17:27:30.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511' 2024-09-10T17:27:30.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:30.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:27:30.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672989 2024-09-10T17:27:30.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672989 2024-09-10T17:27:30.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-42949672989' 2024-09-10T17:27:30.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:30.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:27:31.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-10T17:27:31.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-10T17:27:31.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-42949672989 2-64424509459' 2024-09-10T17:27:31.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:31.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836511 2024-09-10T17:27:31.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:31.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:27:31.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836511 2024-09-10T17:27:31.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:31.014 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836511 2024-09-10T17:27:31.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836511 2024-09-10T17:27:31.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-10T17:27:31.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:27:31.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836511 2024-09-10T17:27:31.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:27:32.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:27:32.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:27:32.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836511 2024-09-10T17:27:32.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:32.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672989 2024-09-10T17:27:32.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:32.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:27:32.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672989 2024-09-10T17:27:32.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:32.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672989 2024-09-10T17:27:32.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-10T17:27:32.834 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672989 2024-09-10T17:27:32.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:27:33.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672989 2024-09-10T17:27:33.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:33.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-10T17:27:33.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:33.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:27:33.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-10T17:27:33.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:33.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-10T17:27:33.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-10T17:27:33.215 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509459 2024-09-10T17:27:33.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:27:33.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-10T17:27:33.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:27:33.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-09-10T17:27:33.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:27:33.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:27:33.643 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:33.643 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:33.643 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:27:33.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-osd.0.asok 2024-09-10T17:27:33.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:27:33.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-osd.0.asok dump_osd_network 2024-09-10T17:27:33.721 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:27:33.721 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-10T17:27:33.721 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-10T17:27:33.721 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:27:33.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:33.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:27:33.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:27:33.732 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:33.732 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-09-10T17:27:33.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-09-10T17:27:33.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:27:33.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-10T17:27:33.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-10T17:27:33.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-10T17:27:33.736 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:33.736 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:33.736 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:27:33.736 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mgr.x.asok 2024-09-10T17:27:33.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:27:33.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-mgr.x.asok dump_osd_network 2024-09-10T17:27:33.807 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:27:33.807 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-10T17:27:33.807 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-10T17:27:33.807 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:27:33.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:33.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:27:33.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:27:33.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:33.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-09-10T17:27:33.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-09-10T17:27:33.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:27:33.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-09-10T17:27:33.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:27:33.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:27:33.821 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:33.821 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:33.822 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:27:33.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-osd.0.asok 2024-09-10T17:27:33.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:27:33.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-osd.0.asok dump_osd_network 0 2024-09-10T17:27:33.900 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:27:33.900 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-10T17:27:33.900 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-10T17:27:33.900 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.900 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:31 2024", 2024-09-10T17:27:33.900 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.900 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:33.900 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.304, 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.336, 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.324 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.360, 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.457, 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.479 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.360, 2024-09-10T17:27:33.901 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.457, 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.479 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.239 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:31 2024", 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.288, 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.316, 2024-09-10T17:27:33.902 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.310 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.374, 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.488, 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.493 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.374, 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.488, 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.493 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.180 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.903 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:30 2024", 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.286, 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.306, 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.309 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.904 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.339, 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.428, 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.449 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.339, 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.428, 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.449 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.213 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:30 2024", 2024-09-10T17:27:33.905 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.260, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.296, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.294 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.318, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.442, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.519 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.906 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.907 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.318, 2024-09-10T17:27:33.907 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.442, 2024-09-10T17:27:33.907 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.519 2024-09-10T17:27:33.907 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.907 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.200 2024-09-10T17:27:33.907 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-10T17:27:33.907 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-10T17:27:33.907 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:27:33.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:33.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:27:33.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-09-10T17:27:33.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:33.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-09-10T17:27:33.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:27:33.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:27:33.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-10T17:27:33.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-10T17:27:33.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-10T17:27:33.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:33.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:33.916 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:27:33.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mgr.x.asok 2024-09-10T17:27:33.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:27:33.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-mgr.x.asok dump_osd_network 0 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:26 2024", 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:33.986 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.336, 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.351, 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.324 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.296, 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.200, 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.200 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.987 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.437, 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.469, 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.479 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.217 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:29 2024", 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.288, 2024-09-10T17:27:33.988 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.314, 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.344 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.252, 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.235, 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.235 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.335, 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.440, 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.503 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.989 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.208 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:29 2024", 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.268, 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.310, 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.336 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.990 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.236, 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.193, 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.193 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.297, 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.451, 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.498 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.237 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:26 2024", 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.991 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.290, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.325, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.310 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.234, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.214, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.208 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.356, 2024-09-10T17:27:33.992 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.488, 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.493 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.198 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:29 2024", 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.285, 2024-09-10T17:27:33.993 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.302, 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.318 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.252, 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.222, 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.206 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.336, 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.460, 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.476 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.224 2024-09-10T17:27:33.994 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:30 2024", 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.286, 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.306, 2024-09-10T17:27:33.995 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.309 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.231, 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.173, 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.173 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.339, 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.428, 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.449 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.231 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.996 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:30 2024", 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.297, 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.307, 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.302 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.203, 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.203, 2024-09-10T17:27:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.203 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.414, 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.414, 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.485 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.203 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:29 2024", 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:27:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.257, 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.280, 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.303 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.212, 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.206, 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.186 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.314, 2024-09-10T17:27:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.438, 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.445 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.253 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:30 2024", 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.260, 2024-09-10T17:27:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.296, 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.294 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.185, 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.167, 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.164 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.318, 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.442, 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.519 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.185 2024-09-10T17:27:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:30 2024", 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.294, 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.289, 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.284 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.229, 2024-09-10T17:27:34.002 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.201, 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.149 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.454, 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.454, 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.525 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.240 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:30 2024", 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:27:34.003 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.283, 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.285, 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.282 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.191, 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.191, 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.187 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:34.004 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.431, 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.454, 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.500 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.191 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:27:30 2024", 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.267, 2024-09-10T17:27:34.005 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.277, 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.272 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.175, 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.175, 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.170 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.396, 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.434, 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.467 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:27:34.006 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.175 2024-09-10T17:27:34.007 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-10T17:27:34.007 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-10T17:27:34.007 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:27:34.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:34.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:27:34.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-09-10T17:27:34.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:27:34.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-09-10T17:27:34.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:27:34.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-09-10T17:28:24.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-09-10T17:28:24.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:28:24.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:28:24.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:28:24.434 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:28:24.434 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-10T17:28:24.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:28:24.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:28:24.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:28:24.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-10T17:28:24.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-10T17:28:24.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-10T17:28:24.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:28:24.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:28:24.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673002 2024-09-10T17:28:24.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673002 2024-09-10T17:28:24.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2024-09-10T17:28:24.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:28:24.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:28:24.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509471 2024-09-10T17:28:24.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509471 2024-09-10T17:28:24.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509471' 2024-09-10T17:28:24.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:28:24.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-10T17:28:24.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:28:24.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:28:24.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-10T17:28:24.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:28:24.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-10T17:28:24.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-10T17:28:24.786 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836524 2024-09-10T17:28:24.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:28:25.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-10T17:28:25.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:28:25.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673002 2024-09-10T17:28:25.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:28:25.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:28:25.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673002 2024-09-10T17:28:25.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:28:25.169 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949673002 2024-09-10T17:28:25.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673002 2024-09-10T17:28:25.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2024-09-10T17:28:25.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:28:25.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673000 -lt 42949673002 2024-09-10T17:28:25.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:28:26.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:28:26.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:28:26.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673003 -lt 42949673002 2024-09-10T17:28:26.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:28:26.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509471 2024-09-10T17:28:26.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:28:26.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:28:26.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509471 2024-09-10T17:28:26.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:28:26.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509471 2024-09-10T17:28:26.986 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509471 2024-09-10T17:28:26.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-10T17:28:26.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:28:27.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509472 -lt 64424509471 2024-09-10T17:28:27.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:28:27.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-09-10T17:28:27.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:28:27.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:28:27.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:27.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:27.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:27.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-osd.0.asok 2024-09-10T17:28:27.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:28:27.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-osd.0.asok dump_osd_network 2024-09-10T17:28:27.494 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:28:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-10T17:28:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-10T17:28:27.495 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:28:27.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:28:27.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:28:27.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-09-10T17:28:27.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-09-10T17:28:27.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mgr.x.asok 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:28:27.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-mgr.x.asok dump_osd_network 2024-09-10T17:28:27.581 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:28:27.581 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-10T17:28:27.581 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-10T17:28:27.581 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:28:27.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:28:27.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:28:27.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-09-10T17:28:27.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-09-10T17:28:27.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:28:27.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-09-10T17:28:27.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:28:27.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:28:27.595 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:27.595 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:27.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:27.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-osd.0.asok 2024-09-10T17:28:27.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:28:27.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-osd.0.asok dump_osd_network 0 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:26 2024", 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:28:27.674 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.284, 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.335, 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.329 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.410, 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.504, 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.504 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.410, 2024-09-10T17:28:27.675 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.504, 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.504 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.266 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:26 2024", 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:28:27.676 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.306, 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.332, 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.319 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.391, 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.446, 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.515 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.677 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.391, 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.446, 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.515 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.234 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:25 2024", 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:28:27.678 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.285, 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.332, 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.313 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.373, 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.478, 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.478 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.373, 2024-09-10T17:28:27.679 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.478, 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.478 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.247 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:25 2024", 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.680 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.285, 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.315, 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.300 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.355, 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.514, 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.514 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.355, 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.514, 2024-09-10T17:28:27.681 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.514 2024-09-10T17:28:27.682 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.682 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.218 2024-09-10T17:28:27.682 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-10T17:28:27.682 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-10T17:28:27.682 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:28:27.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:28:27.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-09-10T17:28:27.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-09-10T17:28:27.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:28:27.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:28:27.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-10T17:28:27.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-10T17:28:27.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-10T17:28:27.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:27.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:27.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:27.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mgr.x.asok 2024-09-10T17:28:27.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:28:27.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-mgr.x.asok dump_osd_network 0 2024-09-10T17:28:27.758 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:28:27.758 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-10T17:28:27.758 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:21 2024", 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.358, 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.345, 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.319 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.288, 2024-09-10T17:28:27.759 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.249, 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.180 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.441, 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.515, 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.515 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.203 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:21 2024", 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:28:27.760 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.355, 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.351, 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.332 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.253, 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.232, 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.200 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.761 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.455, 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.504, 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.504 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.190 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:25 2024", 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.762 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.285, 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.332, 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.313 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.172, 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.172, 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.172 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.373, 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.478, 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.478 2024-09-10T17:28:27.763 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.172 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:24 2024", 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.287, 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.322, 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.325 2024-09-10T17:28:27.764 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.242, 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.235, 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.208 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.349, 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.465, 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.490 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.242 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.765 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:25 2024", 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.320, 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.325, 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.325 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.264, 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.251, 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.203 2024-09-10T17:28:27.766 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.462, 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.477, 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.522 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.264 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:24 2024", 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:28:27.767 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.287, 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.312, 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.322 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.232, 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.212, 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.212 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.334, 2024-09-10T17:28:27.768 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.502, 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.502 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.253 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:25 2024", 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.314, 2024-09-10T17:28:27.769 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.319, 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.310 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.250, 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.250, 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.184 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.436, 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.458, 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.465 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.250 2024-09-10T17:28:27.770 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:24 2024", 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.268, 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.307, 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.315 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.771 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.168, 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.168, 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.168 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.326, 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.480, 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.507 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.168 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:25 2024", 2024-09-10T17:28:27.772 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.285, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.315, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.300 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.145, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.145, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.145 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.773 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.355, 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.514, 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.514 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.145 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:24 2024", 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.774 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.274, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.295, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.303 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.201, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.201, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.201 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.310, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.422, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.493 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.775 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.201 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:25 2024", 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.268, 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.298, 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.302 2024-09-10T17:28:27.776 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.218, 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.218, 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.181 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.413, 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.433, 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.481 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.218 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-10T17:28:27.777 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Tue Sep 10 17:28:25 2024", 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.264, 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.288, 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.292 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.203, 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.203, 2024-09-10T17:28:27.778 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.175 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.369, 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.434, 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.448 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.203 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-10T17:28:27.779 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:28:27.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-09-10T17:28:27.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-09-10T17:28:27.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-09-10T17:28:27.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-09-10T17:28:27.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:28:27.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-10T17:28:27.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-10T17:28:27.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-10T17:28:27.782 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:27.782 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:27.782 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:27.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mgr.x.asok 2024-09-10T17:28:27.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:28:27.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-mgr.x.asok dump_osd_network 99 2024-09-10T17:28:27.843 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:28:27.844 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 99, 2024-09-10T17:28:27.844 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-10T17:28:27.844 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:28:27.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-09-10T17:28:27.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-09-10T17:28:27.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-osd.0.asok 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-09-10T17:28:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41145/ceph-osd.0.asok dump_osd_network 98 2024-09-10T17:28:27.932 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-10T17:28:27.932 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 98, 2024-09-10T17:28:27.932 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-10T17:28:27.932 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-10T17:28:27.940 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-09-10T17:28:27.940 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-09-10T17:28:27.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-09-10T17:28:27.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-09-10T17:28:27.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-10T17:28:27.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-10T17:28:27.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:28:27.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-10T17:28:27.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:28:27.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:28:27.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:28:27.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:28:27.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:28:28.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:28:28.052 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:28:28.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:28:28.053 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:28:28.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:28:28.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:28:28.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:28:28.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:28:28.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:28:28.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:28:28.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:28:28.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:28:28.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:28:28.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-10T17:28:28.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:28:28.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:28.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:28.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.41145 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:28:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:28:28.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:28:28.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:28:28.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:28:28.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:28:28.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:28:28.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:28:28.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:28:28.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:28:28.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:28:28.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:28:28.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:28:28.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:28:28.084 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:28:28.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:28:28.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:28:28.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-10T17:28:28.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:28:28.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:28.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:28.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.41145 2024-09-10T17:28:28.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:28:28.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:28:28.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2024-09-10T17:28:28.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:28:28.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:28.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:28.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.41145 2024-09-10T17:28:28.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:28:28.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:28:28.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:28:28.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-10T17:28:28.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2024-09-10T17:28:28.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-10T17:28:28.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:28:28.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:28:28.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:28:28.562 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:28:28.562 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:28.562 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:28.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:28:28.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:28:28.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:28:28.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:28:28.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:28:28.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:28:28.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:28:28.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:28:28.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:28:28.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:28:28.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:28:28.748 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:28.748 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:28.748 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:28.748 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mon.a.asok 2024-09-10T17:28:28.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:28:28.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.41145/ceph-mon.a.asok config get fsid 2024-09-10T17:28:28.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:28.831 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.41145/ceph-mon.a.asok 2024-09-10T17:28:28.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:28:28.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.41145/ceph-mon.a.asok config get mon_host 2024-09-10T17:28:28.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-09-10T17:28:28.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2024-09-10T17:28:28.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:28:28.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:28:28.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:28:28.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2024-09-10T17:28:28.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:28:29.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:28:29.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:28:29.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:28:29.759 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:28:29.759 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:29.759 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:29.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:28:29.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:28:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:28:29.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-09-10T17:28:29.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-10T17:28:29.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:28:29.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:28:29.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:28:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:28:29.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:28:29.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:28:29.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:28:29.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:28:29.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2024-09-10T17:28:29.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:28:29.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ea4d4054-10c8-469a-bb1b-a5194638f111 2024-09-10T17:28:29.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ea4d4054-10c8-469a-bb1b-a5194638f111' 2024-09-10T17:28:29.795 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 ea4d4054-10c8-469a-bb1b-a5194638f111 2024-09-10T17:28:29.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:28:29.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC9geBmnaVGMBAAirQQv5uYil10A8neC8C/kA== 2024-09-10T17:28:29.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC9geBmnaVGMBAAirQQv5uYil10A8neC8C/kA=="}' 2024-09-10T17:28:29.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ea4d4054-10c8-469a-bb1b-a5194638f111 -i td/network-ping/0/new.json 2024-09-10T17:28:30.087 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:28:30.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2024-09-10T17:28:30.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9geBmnaVGMBAAirQQv5uYil10A8neC8C/kA== --osd-uuid ea4d4054-10c8-469a-bb1b-a5194638f111 2024-09-10T17:28:30.129 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:30.127+0000 7f428ac12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:30.157 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:30.155+0000 7f428ac12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:30.158 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:30.156+0000 7f428ac12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:30.158 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:30.156+0000 7f428ac12640 -1 bdev(0x55b8489dfc00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:28:30.158 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:30.156+0000 7f428ac12640 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-10T17:28:35.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-10T17:28:35.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:28:35.789 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:28:35.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:28:35.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:28:36.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:28:36.343 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:28:36.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:28:36.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:28:36.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:28:36.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:28:36.371 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:36.369+0000 7f22c0431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:36.386 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:36.384+0000 7f22c0431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:36.387 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:36.385+0000 7f22c0431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:28:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:28:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:28:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:28:36.791 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:28:36.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:28:36.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:36.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:28:36.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:36.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:37.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:37.757 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:37.756+0000 7f22c0431640 -1 Falling back to public interface 2024-09-10T17:28:38.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:38.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:38.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:28:38.227 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:28:38.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:38.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:38.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:39.338 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:28:39.337+0000 7f22c0431640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:28:39.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:39.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:39.653 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:28:39.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:28:39.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:39.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:40.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:41.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:41.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:41.056 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:28:41.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:28:41.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:41.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:41.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:42.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:42.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:42.467 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:28:42.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:28:42.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:42.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:42.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:43.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:43.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:43.884 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:28:43.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:28:43.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:43.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:44.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:45.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:45.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:45.348 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:28:45.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:28:45.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:45.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:45.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:46.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:46.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:46.779 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:28:46.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:28:46.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:46.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:47.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:48.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:48.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:48.202 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:28:48.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:28:48.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:48.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:48.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:49.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:49.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:49.599 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:28:49.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:28:49.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:49.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:50.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:51.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:51.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:51.110 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:28:51.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:28:51.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:51.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:51.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:52.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:52.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:52.853 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:28:52.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:28:52.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:52.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:53.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:54.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:54.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:54.355 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:28:54.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:28:54.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:54.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:55.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:56.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:56.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:56.149 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:28:56.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:28:56.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:56.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:57.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:58.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:58.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:58.288 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:28:58.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:28:58.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:58.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:59.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:00.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:00.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:00.363 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:29:00.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:29:00.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:00.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:29:01.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:02.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:02.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:02.267 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:29:02.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:29:02.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:02.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:29:02.747 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/921956631,v1:127.0.0.1:6803/921956631] [v2:127.0.0.1:6804/921956631,v1:127.0.0.1:6805/921956631] exists,up ea4d4054-10c8-469a-bb1b-a5194638f111 2024-09-10T17:29:02.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:29:02.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:29:02.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:29:02.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-09-10T17:29:02.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-10T17:29:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:29:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:29:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:29:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2024-09-10T17:29:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-10T17:29:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:29:02.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:29:02.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:29:02.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-10T17:29:02.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-10T17:29:02.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:29:02.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:29:02.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:29:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:29:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:29:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:29:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-10T17:29:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-10T17:29:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:29:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:29:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:29:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:29:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:29:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:29:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2024-09-10T17:29:02.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:29:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=17707dda-7a8b-4647-9bcb-7e12848a4190 2024-09-10T17:29:02.761 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 17707dda-7a8b-4647-9bcb-7e12848a4190 2024-09-10T17:29:02.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 17707dda-7a8b-4647-9bcb-7e12848a4190' 2024-09-10T17:29:02.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:29:02.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDegeBm6yTZLRAAoMFAoM4T+vpcTuM4rZW87w== 2024-09-10T17:29:02.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDegeBm6yTZLRAAoMFAoM4T+vpcTuM4rZW87w=="}' 2024-09-10T17:29:02.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 17707dda-7a8b-4647-9bcb-7e12848a4190 -i td/network-ping/1/new.json 2024-09-10T17:29:03.252 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:29:03.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2024-09-10T17:29:03.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDegeBm6yTZLRAAoMFAoM4T+vpcTuM4rZW87w== --osd-uuid 17707dda-7a8b-4647-9bcb-7e12848a4190 2024-09-10T17:29:03.295 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:03.293+0000 7f4e62e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:03.304 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:03.303+0000 7f4e62e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:03.306 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:03.305+0000 7f4e62e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:03.306 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:03.305+0000 7f4e62e44640 -1 bdev(0x55f33a481c00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:29:03.306 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:03.305+0000 7f4e62e44640 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-10T17:29:09.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-10T17:29:09.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:29:09.099 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-10T17:29:09.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:29:09.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:29:09.632 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-10T17:29:09.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:29:09.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:29:09.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:29:09.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:29:09.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:29:09.661 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:09.660+0000 7fe6f5efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:09.696 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:09.695+0000 7fe6f5efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:09.697 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:09.696+0000 7fe6f5efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:10.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:29:10.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:29:10.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:29:10.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:29:10.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:29:10.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:10.070 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:29:10.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:29:10.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:10.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:10.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:11.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:11.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:11.497 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:29:11.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:29:11.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:11.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:11.554 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:11.553+0000 7fe6f5efe640 -1 Falling back to public interface 2024-09-10T17:29:11.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:12.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:12.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:12.948 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:29:12.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:29:12.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:12.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:13.113 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:13.111+0000 7fe6f5efe640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:29:13.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:14.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:14.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:14.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:29:14.424 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:29:14.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:14.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:14.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:15.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:15.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:15.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:29:15.958 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:29:15.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:15.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:16.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:17.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:17.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:17.490 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:29:17.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:29:17.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:17.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:18.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:19.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:19.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:19.030 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:29:19.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:29:19.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:19.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:19.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:20.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:20.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:20.427 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:29:20.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:29:20.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:20.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:20.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:21.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:21.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:21.944 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:29:21.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:29:21.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:21.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:22.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:23.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:23.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:23.369 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:29:23.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:29:23.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:23.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:23.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:24.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:24.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:24.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:29:24.769 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:29:24.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:24.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:25.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:26.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:26.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:26.411 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:29:26.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:29:26.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:26.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:27.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:28.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:28.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:28.147 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:29:28.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:29:28.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:28.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:29.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:30.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:30.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:30.228 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:29:30.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:29:30.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:30.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:31.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:32.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:32.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:32.314 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:29:32.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:29:32.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:33.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:34.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:34.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:34.377 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:29:34.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:29:34.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:34.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:35.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:36.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:36.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:36.401 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:29:36.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:29:36.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:36.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:36.835 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1811291174,v1:127.0.0.1:6811/1811291174] [v2:127.0.0.1:6812/1811291174,v1:127.0.0.1:6813/1811291174] exists,up 17707dda-7a8b-4647-9bcb-7e12848a4190 2024-09-10T17:29:36.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:29:36.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:29:36.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:29:36.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2024-09-10T17:29:36.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:29:36.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a038bba1-efe1-42aa-8b78-d785b37ac366 2024-09-10T17:29:36.842 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 a038bba1-efe1-42aa-8b78-d785b37ac366 2024-09-10T17:29:36.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a038bba1-efe1-42aa-8b78-d785b37ac366' 2024-09-10T17:29:36.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:29:36.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAAguBmlCoPMxAATNXFc8NGZX8IHyt634eoyQ== 2024-09-10T17:29:36.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAAguBmlCoPMxAATNXFc8NGZX8IHyt634eoyQ=="}' 2024-09-10T17:29:36.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a038bba1-efe1-42aa-8b78-d785b37ac366 -i td/network-ping/2/new.json 2024-09-10T17:29:37.343 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:29:37.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2024-09-10T17:29:37.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAguBmlCoPMxAATNXFc8NGZX8IHyt634eoyQ== --osd-uuid a038bba1-efe1-42aa-8b78-d785b37ac366 2024-09-10T17:29:37.387 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:37.385+0000 7f19eac9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:37.444 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:37.443+0000 7f19eac9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:37.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:37.444+0000 7f19eac9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:37.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:37.444+0000 7f19eac9c640 -1 bdev(0x56191bf61c00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:29:37.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:37.444+0000 7f19eac9c640 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-10T17:29:43.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-10T17:29:43.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:29:43.426 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-10T17:29:43.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:29:43.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:29:43.981 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-10T17:29:43.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:29:43.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=43f2cc8b-6811-4b73-9889-e063f9ac2336 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41145/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:29:43.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:29:43.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:29:43.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:29:44.010 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:44.009+0000 7f2db240f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:44.026 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:44.025+0000 7f2db240f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:44.027 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:44.026+0000 7f2db240f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:44.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:29:44.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:29:44.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:29:44.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:29:44.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:29:44.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:44.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:29:44.402 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:29:44.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:44.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:44.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:45.131 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:45.130+0000 7f2db240f640 -1 Falling back to public interface 2024-09-10T17:29:45.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:45.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:45.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:29:45.851 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:29:45.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:45.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:46.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:47.049 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:29:47.048+0000 7f2db240f640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:29:47.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:47.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:47.252 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:29:47.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:29:47.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:47.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:47.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:48.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:48.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:48.787 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:29:48.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:29:48.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:48.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:49.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:50.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:50.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:50.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:29:50.191 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:29:50.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:50.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:50.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:51.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:51.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:51.731 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:29:51.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:29:51.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:51.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:52.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:53.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:53.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:53.264 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:29:53.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:29:53.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:53.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:53.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:54.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:54.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:54.676 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:29:54.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:29:54.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:54.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:55.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:56.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:56.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:56.116 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:29:56.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:29:56.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:56.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:56.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:57.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:57.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:57.506 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:29:57.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:29:57.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:57.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:58.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:59.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:59.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:59.054 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:29:59.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:29:59.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:59.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:59.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:00.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:00.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:00.750 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:30:00.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:30:00.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:00.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:30:01.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:02.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:02.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:02.779 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:30:02.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:30:02.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:02.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:30:03.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:04.280 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:30:04.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:30:04.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:04.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:30:05.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:06.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:06.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:06.082 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:30:06.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:30:06.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:06.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:30:07.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:08.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:08.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:08.129 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:30:08.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:30:08.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:08.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:30:08.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:09.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:09.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:09.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:30:09.980 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:30:09.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:09.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:30:10.377 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2008614381,v1:127.0.0.1:6819/2008614381] [v2:127.0.0.1:6820/2008614381,v1:127.0.0.1:6821/2008614381] exists,up a038bba1-efe1-42aa-8b78-d785b37ac366 2024-09-10T17:30:10.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:30:10.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:30:10.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:30:10.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-09-10T17:30:15.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-09-10T17:30:15.867 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'dc1' already exists 2024-09-10T17:30:15.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-09-10T17:30:16.375 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'dc2' already exists 2024-09-10T17:30:16.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-09-10T17:30:16.874 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'dc3' already exists 2024-09-10T17:30:16.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-09-10T17:30:17.376 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'rack1' already exists 2024-09-10T17:30:17.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-09-10T17:30:17.917 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'rack2' already exists 2024-09-10T17:30:17.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-09-10T17:30:18.430 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'rack3' already exists 2024-09-10T17:30:18.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-09-10T17:30:18.922 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'host1' already exists 2024-09-10T17:30:18.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-09-10T17:30:19.492 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'host2' already exists 2024-09-10T17:30:19.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-09-10T17:30:20.080 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'host3' already exists 2024-09-10T17:30:20.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-09-10T17:30:20.574 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-09-10T17:30:20.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-09-10T17:30:21.068 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-09-10T17:30:21.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-09-10T17:30:21.719 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-09-10T17:30:21.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-09-10T17:30:22.256 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-09-10T17:30:22.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-09-10T17:30:22.824 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-09-10T17:30:22.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-09-10T17:30:23.321 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-09-10T17:30:23.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-09-10T17:30:23.824 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-09-10T17:30:23.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-09-10T17:30:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-09-10T17:30:24.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-09-10T17:30:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-09-10T17:30:24.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-09-10T17:30:25.353 INFO:tasks.workunit.client.0.smithi155.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-09-10T17:30:25.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-09-10T17:30:25.863 INFO:tasks.workunit.client.0.smithi155.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-09-10T17:30:25.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-09-10T17:30:26.386 INFO:tasks.workunit.client.0.smithi155.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-09-10T17:30:26.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-09-10T17:30:26.874 INFO:tasks.workunit.client.0.smithi155.stderr:rule myrule already exists 2024-09-10T17:30:26.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-09-10T17:30:26.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-09-10T17:30:27.369 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'foo' already exists 2024-09-10T17:30:27.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:30:28.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-10T17:30:28.416 INFO:tasks.workunit.client.0.smithi155.stdout:hints = 1 2024-09-10T17:30:28.416 INFO:tasks.workunit.client.0.smithi155.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-10T17:30:28.416 INFO:tasks.workunit.client.0.smithi155.stdout:Object prefix: benchmark_data_smithi155_53307 2024-09-10T17:30:28.416 INFO:tasks.workunit.client.0.smithi155.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T17:30:28.416 INFO:tasks.workunit.client.0.smithi155.stdout: 0 0 0 0 0 0 - 0 2024-09-10T17:30:29.416 INFO:tasks.workunit.client.0.smithi155.stdout: 1 16 66 50 0.195294 0.195312 0.234862 0.287864 2024-09-10T17:30:30.416 INFO:tasks.workunit.client.0.smithi155.stdout: 2 16 121 105 0.20506 0.214844 0.250254 0.281264 2024-09-10T17:30:31.416 INFO:tasks.workunit.client.0.smithi155.stdout: 3 16 185 169 0.220031 0.25 0.233636 0.268535 2024-09-10T17:30:32.416 INFO:tasks.workunit.client.0.smithi155.stdout: 4 16 258 242 0.236305 0.285156 0.21631 0.260653 2024-09-10T17:30:33.417 INFO:tasks.workunit.client.0.smithi155.stdout: 5 16 322 306 0.239037 0.25 0.220473 0.255022 2024-09-10T17:30:34.417 INFO:tasks.workunit.client.0.smithi155.stdout: 6 16 386 370 0.240858 0.25 0.236219 0.253875 2024-09-10T17:30:35.417 INFO:tasks.workunit.client.0.smithi155.stdout: 7 16 450 434 0.242159 0.25 0.264742 0.253876 2024-09-10T17:30:36.417 INFO:tasks.workunit.client.0.smithi155.stdout: 8 16 514 498 0.243135 0.25 0.235067 0.254457 2024-09-10T17:30:37.417 INFO:tasks.workunit.client.0.smithi155.stdout: 9 16 586 570 0.247365 0.28125 0.200136 0.248867 2024-09-10T17:30:38.417 INFO:tasks.workunit.client.0.smithi155.stdout: 10 16 658 642 0.250751 0.28125 0.22205 0.246875 2024-09-10T17:30:39.417 INFO:tasks.workunit.client.0.smithi155.stdout:Total time run: 10.1699 2024-09-10T17:30:39.417 INFO:tasks.workunit.client.0.smithi155.stdout:Total writes made: 658 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Write size: 4096 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Object size: 4096 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Bandwidth (MB/sec): 0.252737 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev Bandwidth: 0.0286931 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Max bandwidth (MB/sec): 0.285156 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Min bandwidth (MB/sec): 0.195312 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Average IOPS: 64 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev IOPS: 7.34544 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Max IOPS: 73 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Min IOPS: 50 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Average Latency(s): 0.246752 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev Latency(s): 0.0427472 2024-09-10T17:30:39.418 INFO:tasks.workunit.client.0.smithi155.stdout:Max latency(s): 0.538797 2024-09-10T17:30:39.419 INFO:tasks.workunit.client.0.smithi155.stdout:Min latency(s): 0.177096 2024-09-10T17:30:39.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-09-10T17:30:49.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-09-10T17:30:49.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:30:49.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:30:49.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:30:49.886 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:30:49.886 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-10T17:30:49.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:30:49.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:30:49.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:30:50.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-10T17:30:50.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-10T17:30:50.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-10T17:30:50.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:30:50.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:30:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-10T17:30:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-10T17:30:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672978' 2024-09-10T17:30:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:30:50.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:30:50.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-10T17:30:50.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-10T17:30:50.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672978 2-64424509454' 2024-09-10T17:30:50.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:30:50.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-10T17:30:50.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:30:50.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:30:50.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-10T17:30:50.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:30:50.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-10T17:30:50.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-10T17:30:50.252 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836508 2024-09-10T17:30:50.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:30:50.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-10T17:30:50.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:30:51.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:30:51.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:30:52.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-10T17:30:52.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:30:52.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-10T17:30:52.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:30:52.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:30:52.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-10T17:30:52.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:30:52.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-10T17:30:52.076 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672978 2024-09-10T17:30:52.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-10T17:30:52.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:30:52.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672978 2024-09-10T17:30:52.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:30:52.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-10T17:30:52.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:30:52.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:30:52.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-10T17:30:52.489 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:30:52.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-10T17:30:52.490 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509454 2024-09-10T17:30:52.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-10T17:30:52.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:30:52.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509454 2024-09-10T17:30:52.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-09-10T17:30:52.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-09-10T17:30:53.367 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.344ms); Slow OSD heartbeats on front (longest 0.369ms); 1 pool(s) do not have an application enabled 2024-09-10T17:30:53.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-09-10T17:30:53.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-09-10T17:30:53.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-09-10T17:30:53.897 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.344ms); Slow OSD heartbeats on front (longest 0.369ms); 1 pool(s) do not have an application enabled 2024-09-10T17:30:53.897 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.344ms) 2024-09-10T17:30:53.897 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.344 msec 2024-09-10T17:30:53.897 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.322 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.317 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.317 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.312 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.290 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.369ms) 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.369 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.359 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.334 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.327 msec possibly improving 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.318 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.298 msec 2024-09-10T17:30:53.898 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T17:30:53.899 INFO:tasks.workunit.client.0.smithi155.stdout: application not enabled on pool 'foo' 2024-09-10T17:30:53.899 INFO:tasks.workunit.client.0.smithi155.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T17:30:53.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-09-10T17:30:53.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-09-10T17:30:53.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-09-10T17:30:53.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-09-10T17:30:53.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-10T17:30:53.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-10T17:30:53.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:30:53.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-10T17:30:53.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:30:53.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:30:53.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:30:53.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:30:53.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:30:54.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:30:54.038 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:30:54.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:30:54.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:30:54.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:30:54.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:30:54.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:30:54.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:30:54.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:30:54.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:30:54.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:30:54.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:30:54.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:30:54.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-10T17:30:54.063 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:30:54.063 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:54.063 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:30:54.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.41145 2024-09-10T17:30:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:30:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:30:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T17:30:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/network-ping 0 2024-09-10T17:30:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-10T17:30:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T17:30:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-10T17:30:54.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:30:54.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:30:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:30:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:30:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:30:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:30:54.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:30:54.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:30:54.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:30:54.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:30:54.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:30:54.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:30:54.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:30:54.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:30:54.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:30:54.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:30:54.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:30:54.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T17:30:54.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-10T17:30:54.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:30:54.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:54.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.41145 2024-09-10T17:30:54.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.41145 2024-09-10T17:30:54.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:30:54.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:30:54.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T17:30:54.075 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T17:30:54.076 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T17:30:54.142 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-09-10T17:30:54.142 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-09-10T17:30:54.204 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T17:30:54.210 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T17:30:54.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:30:54.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:30:54.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T17:30:54.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/ok-to-stop 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-09-10T17:30:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none ' 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none ' 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none ' 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-09-10T17:30:54.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T17:30:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-09-10T17:30:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_0_osd 2024-09-10T17:30:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_1_mon_checks 2024-09-10T17:30:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_2_mons_checks 2024-09-10T17:30:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_3_mons_checks 2024-09-10T17:30:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_4_mons_checks 2024-09-10T17:30:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_5_mons_checks' 2024-09-10T17:30:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:30:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:30:54.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:30:54.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:30:54.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:30:54.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:30:54.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:30:54.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:30:54.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:30:54.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:30:54.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:30:54.218 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:30:54.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:30:54.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:30:54.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:30:54.220 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:30:54.220 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:54.220 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:30:54.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:30:54.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:30:54.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:30:54.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-10T17:30:54.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:30:54.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:54.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:30:54.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53672 2024-09-10T17:30:54.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:30:54.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T17:30:54.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T17:30:54.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:30:54.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-10T17:30:54.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-09-10T17:30:54.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-09-10T17:30:54.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-10T17:30:54.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-10T17:30:54.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:30:54.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:30:54.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:30:54.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:30:54.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-10T17:30:54.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-10T17:30:54.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:30:54.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:30:54.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:30:54.703 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:30:54.703 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:54.703 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:30:54.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:30:54.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-10T17:30:54.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:30:54.880 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:30:54.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:30:54.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get fsid 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:30:54.960 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:54.961 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:30:54.961 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:30:54.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:30:54.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get mon_host 2024-09-10T17:30:55.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-09-10T17:30:55.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2024-09-10T17:30:55.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:30:55.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:30:55.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:30:55.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2024-09-10T17:30:55.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:30:55.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:30:55.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:30:55.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:30:55.883 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:30:55.883 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:55.883 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:30:55.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:30:55.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:30:55.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:30:55.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-10T17:30:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:30:55.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8e1d4a02-e257-4965-a364-335fb3ca066c 2024-09-10T17:30:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8e1d4a02-e257-4965-a364-335fb3ca066c' 2024-09-10T17:30:55.917 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 8e1d4a02-e257-4965-a364-335fb3ca066c 2024-09-10T17:30:55.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:30:55.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBPguBmDZ5qNxAAMgRZQl4QUXKlZKrgc1nKIQ== 2024-09-10T17:30:55.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBPguBmDZ5qNxAAMgRZQl4QUXKlZKrgc1nKIQ=="}' 2024-09-10T17:30:55.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8e1d4a02-e257-4965-a364-335fb3ca066c -i td/ok-to-stop/0/new.json 2024-09-10T17:30:56.203 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:30:56.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2024-09-10T17:30:56.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPguBmDZ5qNxAAMgRZQl4QUXKlZKrgc1nKIQ== --osd-uuid 8e1d4a02-e257-4965-a364-335fb3ca066c 2024-09-10T17:30:56.241 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:30:56.239+0000 7f0b92e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:56.272 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:30:56.271+0000 7f0b92e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:56.273 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:30:56.272+0000 7f0b92e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:56.273 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:30:56.272+0000 7f0b92e24640 -1 bdev(0x56255657fc00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:30:56.273 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:30:56.272+0000 7f0b92e24640 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-10T17:31:02.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-10T17:31:02.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:31:02.263 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:31:02.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:31:02.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:31:02.791 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:31:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:31:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:31:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:31:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:31:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:31:02.816 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:02.815+0000 7f71bbc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:02.828 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:02.827+0000 7f71bbc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:02.829 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:02.828+0000 7f71bbc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:03.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:31:03.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:31:03.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:31:03.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:31:03.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:31:03.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:03.234 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:31:03.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:31:03.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:03.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:03.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:03.934 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:03.933+0000 7f71bbc2c640 -1 Falling back to public interface 2024-09-10T17:31:04.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:04.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:04.661 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:31:04.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:31:04.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:04.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:05.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:06.006 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:06.004+0000 7f71bbc2c640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:31:06.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:06.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:06.095 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:31:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:31:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:06.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:07.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:07.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:07.626 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:31:07.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:31:07.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:07.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:08.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:09.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:09.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:09.097 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:31:09.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:31:09.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:09.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:09.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:10.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:10.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:10.497 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:31:10.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:31:10.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:10.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:10.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:11.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:11.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:11.924 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:31:11.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:31:11.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:11.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:12.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:13.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:13.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:13.322 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:31:13.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:31:13.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:13.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:13.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:14.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:14.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:14.787 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:31:14.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:31:14.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:14.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:15.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:16.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:16.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:16.208 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:31:16.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:31:16.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:16.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:16.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:17.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:17.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:17.599 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:31:17.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:31:17.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:17.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:18.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:19.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:19.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:19.299 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:31:19.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:31:19.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:19.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:19.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:20.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:20.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:20.883 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:31:20.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:31:20.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:20.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:21.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:22.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:22.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:22.612 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:31:22.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:31:22.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:22.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:23.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:24.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:24.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:24.675 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:31:24.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:31:24.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:24.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:25.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:26.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:26.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:26.478 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:31:26.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:31:26.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:26.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:27.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:27.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:27.960 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:31:27.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:31:27.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:27.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:28.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:29.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:29.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:29.358 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:31:29.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:31:29.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:29.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:29.743 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/231982239,v1:127.0.0.1:6803/231982239] [v2:127.0.0.1:6804/231982239,v1:127.0.0.1:6805/231982239] exists,up 8e1d4a02-e257-4965-a364-335fb3ca066c 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=a 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/a 2024-09-10T17:31:29.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2024-09-10T17:31:29.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:31:29.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:31:29.746 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:31:29.746 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:29.746 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:29.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:31:29.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-10T17:31:29.769 INFO:tasks.workunit.client.0.smithi155.stdout:starting mds.a at 2024-09-10T17:31:29.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-09-10T17:31:30.459 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'meta' already exists 2024-09-10T17:31:30.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-09-10T17:31:31.095 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'data' already exists 2024-09-10T17:31:31.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-09-10T17:31:31.888 INFO:tasks.workunit.client.0.smithi155.stderr:filesystem 'myfs' already exists 2024-09-10T17:31:31.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-09-10T17:31:36.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-09-10T17:31:37.241 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-10T17:31:37.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-09-10T17:31:37.581 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-10T17:31:37.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-09-10T17:31:38.104 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [dne] 2024-09-10T17:31:38.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-09-10T17:31:38.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2024-09-10T17:31:38.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2024-09-10T17:31:38.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=b 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/b 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:31:38.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-10T17:31:38.153 INFO:tasks.workunit.client.0.smithi155.stdout:starting mds.b at 2024-09-10T17:31:38.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-09-10T17:31:43.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-09-10T17:31:43.679 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [a] 2024-09-10T17:31:43.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-09-10T17:31:44.248 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [b] 2024-09-10T17:31:44.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-09-10T17:31:44.604 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-10T17:31:44.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-09-10T17:31:45.119 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [a,dne1,dne2] 2024-09-10T17:31:45.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-09-10T17:31:45.672 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [b,dne] 2024-09-10T17:31:45.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-09-10T17:31:46.068 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-10T17:31:46.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-09-10T17:31:46.597 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [dne1,dne2] 2024-09-10T17:31:46.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-09-10T17:31:46.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:31:46.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:31:46.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:31:46.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:31:46.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:31:46.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:31:46.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-10T17:31:46.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:31:46.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:31:46.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:31:46.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:31:46.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:31:46.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:31:46.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-10T17:31:46.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:31:46.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:31:46.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:31:46.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:31:46.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:31:46.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:31:46.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:31:46.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:31:46.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:31:46.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:31:46.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:31:46.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:31:46.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:31:46.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:31:46.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:31:46.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:31:46.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:31:46.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:31:46.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:31:46.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:31:46.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:31:46.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:31:46.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:31:46.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:46.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:46.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:31:46.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:31:46.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:31:46.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:31:46.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:31:46.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:31:46.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:31:46.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:31:46.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:31:46.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:31:46.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:31:46.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:31:46.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:31:46.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:31:46.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:31:46.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:31:46.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:31:46.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:31:46.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:31:46.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:31:46.847 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:31:46.848 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:46.848 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:46.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:31:46.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:31:46.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:31:46.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-10T17:31:46.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:31:46.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:46.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:46.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53672 2024-09-10T17:31:46.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:31:46.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:31:46.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:31:46.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-10T17:31:46.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-09-10T17:31:46.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-09-10T17:31:46.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-10T17:31:46.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-10T17:31:46.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:31:46.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:31:46.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:31:46.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:31:46.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-10T17:31:46.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-10T17:31:47.276 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:31:47.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:31:47.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:31:47.277 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:31:47.277 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:47.277 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:47.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:31:47.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-10T17:31:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:31:47.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:31:47.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:31:47.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:31:47.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:31:47.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:31:47.462 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:31:47.462 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:31:47.462 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:31:47.463 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:31:47.463 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:47.463 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:47.463 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:31:47.463 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:31:47.463 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get fsid 2024-09-10T17:31:47.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:31:47.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:31:47.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:31:47.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:31:47.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:31:47.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:31:47.543 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:31:47.543 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:31:47.543 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:31:47.543 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:47.543 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:47.543 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:31:47.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:31:47.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get mon_host 2024-09-10T17:31:47.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-09-10T17:31:47.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2024-09-10T17:31:47.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:31:47.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:31:47.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:31:47.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2024-09-10T17:31:47.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:31:48.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:31:48.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:31:48.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:31:48.457 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:31:48.457 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:48.457 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:48.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:31:48.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:31:48.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:31:48.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-09-10T17:31:48.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:31:48.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:31:48.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2024-09-10T17:31:48.489 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:31:48.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=12b9453a-b612-4f97-aae1-5ee12f1060a4 2024-09-10T17:31:48.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 12b9453a-b612-4f97-aae1-5ee12f1060a4' 2024-09-10T17:31:48.490 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 12b9453a-b612-4f97-aae1-5ee12f1060a4 2024-09-10T17:31:48.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:31:48.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCEguBmSFP4HRAAPPOkCOqcuEbGEAVsa9uqGg== 2024-09-10T17:31:48.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCEguBmSFP4HRAAPPOkCOqcuEbGEAVsa9uqGg=="}' 2024-09-10T17:31:48.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 12b9453a-b612-4f97-aae1-5ee12f1060a4 -i td/ok-to-stop/0/new.json 2024-09-10T17:31:48.793 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:31:48.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2024-09-10T17:31:48.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEguBmSFP4HRAAPPOkCOqcuEbGEAVsa9uqGg== --osd-uuid 12b9453a-b612-4f97-aae1-5ee12f1060a4 2024-09-10T17:31:48.831 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:48.829+0000 7f2c2b817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:48.870 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:48.861+0000 7f2c2b817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:48.870 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:48.862+0000 7f2c2b817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:48.870 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:48.862+0000 7f2c2b817640 -1 bdev(0x555a0a70bc00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:31:48.871 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:48.862+0000 7f2c2b817640 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-10T17:31:54.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-10T17:31:54.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:31:54.876 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:31:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:31:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:31:55.410 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:31:55.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:31:55.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:31:55.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:31:55.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:31:55.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:31:55.435 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:55.434+0000 7faa41a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:55.447 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:55.446+0000 7faa41a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:55.448 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:55.447+0000 7faa41a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:55.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:31:55.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:31:55.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:31:55.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:31:55.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:31:55.845 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:31:55.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:55.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:31:55.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:55.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:56.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:57.068 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:57.066+0000 7faa41a0f640 -1 Falling back to public interface 2024-09-10T17:31:57.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:57.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:57.272 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:31:57.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:31:57.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:57.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:57.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:58.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:58.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:58.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:31:58.697 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:31:58.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:58.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:31:58.882 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:31:58.881+0000 7faa41a0f640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:31:59.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:00.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:00.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:00.150 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:32:00.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:32:00.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:00.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:00.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:01.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:01.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:01.623 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:32:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:32:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:02.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:03.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:03.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:03.051 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:32:03.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:32:03.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:03.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:03.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:04.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:04.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:04.443 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:32:04.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:32:04.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:04.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:04.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:05.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:05.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:05.995 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:32:05.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:32:05.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:05.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:06.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:07.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:07.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:07.386 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:32:07.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:32:07.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:07.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:07.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:08.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:08.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:08.814 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:32:08.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:32:08.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:08.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:09.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:10.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:10.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:10.222 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:32:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:32:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:11.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:12.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:12.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:12.066 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:32:12.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:32:12.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:12.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:12.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:13.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:13.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:13.778 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:32:13.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:32:13.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:13.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:15.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:15.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:15.463 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:32:15.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:32:15.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:15.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:16.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:17.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:17.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:17.232 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:32:17.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:32:17.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:17.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:18.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:19.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:19.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:19.078 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:32:19.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:32:19.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:19.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:20.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:21.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:21.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:21.123 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:32:21.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:32:21.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:21.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:21.551 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1648187000,v1:127.0.0.1:6803/1648187000] [v2:127.0.0.1:6804/1648187000,v1:127.0.0.1:6805/1648187000] exists,up 12b9453a-b612-4f97-aae1-5ee12f1060a4 2024-09-10T17:32:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:32:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:32:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/1 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-09-10T17:32:21.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:32:21.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/1 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5fdd5ac6-55e4-4dcb-b4d3-70e2fb030332 2024-09-10T17:32:21.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5fdd5ac6-55e4-4dcb-b4d3-70e2fb030332' 2024-09-10T17:32:21.560 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 5fdd5ac6-55e4-4dcb-b4d3-70e2fb030332 2024-09-10T17:32:21.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:32:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQClguBmvwD7IRAAz41E/3RcDMXzzGWaaDde4g== 2024-09-10T17:32:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQClguBmvwD7IRAAz41E/3RcDMXzzGWaaDde4g=="}' 2024-09-10T17:32:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5fdd5ac6-55e4-4dcb-b4d3-70e2fb030332 -i td/ok-to-stop/1/new.json 2024-09-10T17:32:22.004 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:32:22.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/1/new.json 2024-09-10T17:32:22.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQClguBmvwD7IRAAz41E/3RcDMXzzGWaaDde4g== --osd-uuid 5fdd5ac6-55e4-4dcb-b4d3-70e2fb030332 2024-09-10T17:32:22.043 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:22.042+0000 7f2718015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:22.045 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:22.044+0000 7f2718015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:22.046 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:22.045+0000 7f2718015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:22.046 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:22.045+0000 7f2718015640 -1 bdev(0x5649873f9c00 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:32:22.046 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:22.045+0000 7f2718015640 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-09-10T17:32:27.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-09-10T17:32:27.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:32:27.820 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-10T17:32:27.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:32:27.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:32:28.353 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-10T17:32:28.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:32:28.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:32:28.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:32:28.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:32:28.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:32:28.378 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:28.376+0000 7f299ee18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:28.401 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:28.399+0000 7f299ee18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:28.402 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:28.401+0000 7f299ee18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:32:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:32:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:32:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:32:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:32:28.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:28.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:32:28.783 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:32:28.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:28.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:29.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:29.995 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:29.994+0000 7f299ee18640 -1 Falling back to public interface 2024-09-10T17:32:30.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:30.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:30.200 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:32:30.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:32:30.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:30.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:30.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:31.611 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:31.610+0000 7f299ee18640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:32:31.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:31.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:31.626 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:32:31.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:32:31.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:31.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:32.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:33.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:33.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:33.083 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:32:33.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:32:33.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:33.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:33.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:34.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:34.518 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:32:34.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:34.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:32:34.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:34.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:34.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:35.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:35.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:35.929 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:32:35.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:32:35.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:35.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:36.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:37.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:37.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:37.374 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:32:37.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:32:37.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:37.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:38.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:38.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:38.786 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:32:38.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:32:38.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:38.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:39.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:40.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:40.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:40.190 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:32:40.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:32:40.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:40.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:40.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:41.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:41.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:41.580 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:32:41.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:32:41.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:41.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:41.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:42.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:42.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:42.968 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:32:42.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:32:42.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:42.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:43.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:44.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:44.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:44.567 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:32:44.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:32:44.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:44.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:45.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:46.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:46.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:46.630 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:32:46.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:32:46.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:46.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:47.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:48.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:48.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:48.530 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:32:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:32:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:49.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:50.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:50.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:50.819 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:32:50.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:32:50.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:50.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:51.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:52.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:52.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:52.613 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:32:52.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:32:52.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:52.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:53.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:54.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:54.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:54.514 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:32:54.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:32:54.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:54.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:32:54.937 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2079746598,v1:127.0.0.1:6811/2079746598] [v2:127.0.0.1:6812/2079746598,v1:127.0.0.1:6813/2079746598] exists,up 5fdd5ac6-55e4-4dcb-b4d3-70e2fb030332 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:32:54.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/2 2024-09-10T17:32:54.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-10T17:32:54.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:32:54.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:32:54.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:32:54.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-09-10T17:32:54.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-09-10T17:32:54.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:32:54.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:32:54.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:32:54.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:32:54.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-10T17:32:54.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-10T17:32:54.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:32:54.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:32:54.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:32:54.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:32:54.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:32:54.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:32:54.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/2 2024-09-10T17:32:54.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:32:54.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6c1f1672-cbf8-42cc-8341-ebe6f84f0cad 2024-09-10T17:32:54.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6c1f1672-cbf8-42cc-8341-ebe6f84f0cad' 2024-09-10T17:32:54.946 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 6c1f1672-cbf8-42cc-8341-ebe6f84f0cad 2024-09-10T17:32:54.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:32:54.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDGguBmd00IORAAS+ydLAXN6geLb3Xu43/uNQ== 2024-09-10T17:32:54.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDGguBmd00IORAAS+ydLAXN6geLb3Xu43/uNQ=="}' 2024-09-10T17:32:54.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6c1f1672-cbf8-42cc-8341-ebe6f84f0cad -i td/ok-to-stop/2/new.json 2024-09-10T17:32:55.417 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:32:55.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/2/new.json 2024-09-10T17:32:55.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGguBmd00IORAAS+ydLAXN6geLb3Xu43/uNQ== --osd-uuid 6c1f1672-cbf8-42cc-8341-ebe6f84f0cad 2024-09-10T17:32:55.456 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:55.455+0000 7fadde811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:55.459 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:55.457+0000 7fadde811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:55.460 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:55.459+0000 7fadde811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:55.460 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:55.459+0000 7fadde811640 -1 bdev(0x5593e7687c00 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:32:55.460 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:32:55.459+0000 7fadde811640 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-09-10T17:33:01.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-09-10T17:33:01.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:33:01.486 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-10T17:33:01.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:33:01.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:33:02.017 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-10T17:33:02.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:33:02.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:33:02.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:33:02.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:33:02.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:33:02.042 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:02.041+0000 7f263c330640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:02.061 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:02.060+0000 7f263c330640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:02.062 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:02.061+0000 7f263c330640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:02.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:33:02.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:33:02.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:33:02.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:33:02.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:33:02.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:02.468 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:33:02.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:33:02.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:02.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:02.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:02.915 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:02.914+0000 7f263c330640 -1 Falling back to public interface 2024-09-10T17:33:03.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:03.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:03.886 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:33:03.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:33:03.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:03.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:04.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:04.771 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:04.770+0000 7f263c330640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:33:05.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:05.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:05.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:33:05.279 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:33:05.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:05.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:05.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:06.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:06.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:06.787 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:33:06.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:33:06.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:06.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:07.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:08.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:08.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:08.208 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:33:08.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:33:08.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:08.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:08.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:09.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:09.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:09.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:33:09.672 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:33:09.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:09.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:10.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:11.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:11.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:11.086 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:33:11.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:33:11.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:11.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:11.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:12.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:12.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:12.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:33:12.605 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:33:12.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:12.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:13.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:14.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:14.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:14.016 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:33:14.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:33:14.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:14.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:14.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:15.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:15.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:15.414 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:33:15.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:33:15.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:15.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:15.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:16.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:16.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:16.899 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:33:16.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:33:16.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:16.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:17.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:18.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:18.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:18.998 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:33:18.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:33:18.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:18.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:19.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:20.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:20.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:20.810 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:33:20.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:33:20.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:20.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:21.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:22.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:22.711 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:33:22.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:22.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:33:22.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:22.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:23.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:24.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:24.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:24.439 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:33:24.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:33:24.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:24.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:25.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:26.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:26.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:26.651 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:33:26.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:33:26.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:26.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:27.050 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2703320799,v1:127.0.0.1:6819/2703320799] [v2:127.0.0.1:6820/2703320799,v1:127.0.0.1:6821/2703320799] exists,up 6c1f1672-cbf8-42cc-8341-ebe6f84f0cad 2024-09-10T17:33:27.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:33:27.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:33:27.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:33:27.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/3 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:33:27.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:33:27.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/3 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=61680d92-df4d-46c2-960a-8d975a42d79e 2024-09-10T17:33:27.057 INFO:tasks.workunit.client.0.smithi155.stdout:add osd3 61680d92-df4d-46c2-960a-8d975a42d79e 2024-09-10T17:33:27.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 61680d92-df4d-46c2-960a-8d975a42d79e' 2024-09-10T17:33:27.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:33:27.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDnguBm1iNBBBAAyDFPeROuMQbBfSnjQ5vhiQ== 2024-09-10T17:33:27.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDnguBm1iNBBBAAyDFPeROuMQbBfSnjQ5vhiQ=="}' 2024-09-10T17:33:27.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 61680d92-df4d-46c2-960a-8d975a42d79e -i td/ok-to-stop/3/new.json 2024-09-10T17:33:27.643 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:33:27.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/3/new.json 2024-09-10T17:33:27.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnguBm1iNBBBAAyDFPeROuMQbBfSnjQ5vhiQ== --osd-uuid 61680d92-df4d-46c2-960a-8d975a42d79e 2024-09-10T17:33:27.683 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:27.682+0000 7f517861a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:27.702 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:27.701+0000 7f517861a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:27.703 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:27.702+0000 7f517861a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:27.704 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:27.702+0000 7f517861a640 -1 bdev(0x5647ef6e5c00 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:33:27.704 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:27.702+0000 7f517861a640 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-09-10T17:33:33.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-09-10T17:33:33.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:33:33.951 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd3 key to auth repository 2024-09-10T17:33:33.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:33:33.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:33:34.489 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.3 2024-09-10T17:33:34.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:33:34.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:33:34.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:33:34.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:33:34.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:33:34.526 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:34.513+0000 7fe2f6920640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:34.536 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:34.533+0000 7fe2f6920640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:34.536 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:34.534+0000 7fe2f6920640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:34.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:33:34.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:33:34.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:33:34.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:33:34.933 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:33:34.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:33:34.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:34.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:33:34.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:34.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:35.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:35.384 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:35.383+0000 7fe2f6920640 -1 Falling back to public interface 2024-09-10T17:33:36.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:36.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:36.351 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:33:36.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:33:36.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:36.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:36.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:37.346 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:33:37.344+0000 7fe2f6920640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:33:37.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:37.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:37.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:33:37.759 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:33:37.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:37.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:38.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:39.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:39.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:39.151 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:33:39.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:33:39.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:39.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:39.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:40.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:40.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:40.573 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:33:40.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:33:40.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:40.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:41.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:42.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:42.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:42.011 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:33:42.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:33:42.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:42.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:42.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:43.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:43.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:43.469 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:33:43.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:33:43.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:43.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:43.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:44.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:44.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:44.869 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:33:44.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:33:44.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:44.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:45.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:46.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:46.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:46.329 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:33:46.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:33:46.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:46.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:46.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:47.726 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:33:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:33:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:48.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:49.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:49.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:49.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:33:49.137 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:33:49.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:49.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:49.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:50.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:50.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:50.766 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:33:50.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:33:50.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:50.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:51.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:52.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:52.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:52.914 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:33:52.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:33:52.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:52.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:53.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:54.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:54.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:54.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:33:54.599 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:33:54.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:54.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:55.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:56.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:56.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:56.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:33:56.476 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:33:56.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:56.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:56.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:57.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:57.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:57.931 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:33:57.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:33:57.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:57.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:33:58.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:59.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:59.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:59.681 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:33:59.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:33:59.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:59.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:00.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:01.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:01.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:01.075 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:34:01.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:34:01.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:01.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:01.506 INFO:tasks.workunit.client.0.smithi155.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2047460297,v1:127.0.0.1:6827/2047460297] [v2:127.0.0.1:6828/2047460297,v1:127.0.0.1:6829/2047460297] exists,up 61680d92-df4d-46c2-960a-8d975a42d79e 2024-09-10T17:34:01.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:34:01.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:34:01.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:34:01.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-10T17:34:02.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-09-10T17:34:02.698 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'ec' already exists 2024-09-10T17:34:02.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-09-10T17:34:02.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:34:02.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:34:02.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:34:02.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:34:02.711 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:34:02.711 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:34:02.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:34:02.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:34:02.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:34:02.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:34:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:34:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:34:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:34:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:34:02.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:34:03.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:34:03.616 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:34:03.616 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-10T17:34:03.616 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2024-09-10T17:34:03.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:34:03.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:03.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:34:03.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-10T17:34:03.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-10T17:34:03.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-10T17:34:03.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:03.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:34:04.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-10T17:34:04.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-10T17:34:04.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672979' 2024-09-10T17:34:04.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:04.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:34:04.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-10T17:34:04.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-10T17:34:04.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672979 2-64424509455' 2024-09-10T17:34:04.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:04.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:34:04.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345926 2024-09-10T17:34:04.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345926 2024-09-10T17:34:04.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672979 2-64424509455 3-85899345926' 2024-09-10T17:34:04.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:04.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-10T17:34:04.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:04.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:34:04.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-10T17:34:04.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:04.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-10T17:34:04.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-10T17:34:04.484 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836510 2024-09-10T17:34:04.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:34:05.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-10T17:34:05.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:05.016 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-10T17:34:05.016 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:05.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:34:05.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-10T17:34:05.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:05.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-10T17:34:05.019 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672979 2024-09-10T17:34:05.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-10T17:34:05.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:34:05.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-10T17:34:05.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:05.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-10T17:34:05.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:05.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:34:05.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-10T17:34:05.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:05.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-10T17:34:05.664 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509455 2024-09-10T17:34:05.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-10T17:34:05.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:34:06.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-10T17:34:06.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:06.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345926 2024-09-10T17:34:06.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:06.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:34:06.159 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345926 2024-09-10T17:34:06.159 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:06.160 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 85899345926 2024-09-10T17:34:06.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345926 2024-09-10T17:34:06.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-10T17:34:06.160 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:34:06.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345924 -lt 85899345926 2024-09-10T17:34:06.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:34:07.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:34:07.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:34:08.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345927 -lt 85899345926 2024-09-10T17:34:08.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:34:08.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:08.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:08.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-10T17:34:08.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:34:08.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:34:08.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:34:08.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:34:08.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:34:08.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:34:08.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:34:08.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-10T17:34:08.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:34:08.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:08.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:09.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-10T17:34:09.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:34:09.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:34:09.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-09-10T17:34:10.163 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 min_size to 3 2024-09-10T17:34:10.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-09-10T17:34:10.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:34:10.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:34:10.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:34:10.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:34:10.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:34:10.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:34:10.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:34:10.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:34:10.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:34:10.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:34:10.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:34:10.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:34:10.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:34:10.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:34:10.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:34:10.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:34:10.643 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:34:10.643 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-10T17:34:10.643 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2024-09-10T17:34:10.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:34:10.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:10.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:34:10.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-10T17:34:10.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-10T17:34:10.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-10T17:34:10.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:10.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:34:10.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-10T17:34:10.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-10T17:34:10.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672983' 2024-09-10T17:34:10.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:10.870 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:34:10.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-10T17:34:10.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-10T17:34:10.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672983 2-64424509458' 2024-09-10T17:34:10.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:10.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:34:11.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345930 2024-09-10T17:34:11.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345930 2024-09-10T17:34:11.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672983 2-64424509458 3-85899345930' 2024-09-10T17:34:11.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:11.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-10T17:34:11.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:11.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:34:11.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-10T17:34:11.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:11.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-10T17:34:11.094 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836513 2024-09-10T17:34:11.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-10T17:34:11.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:34:11.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836513 2024-09-10T17:34:11.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:34:12.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:34:12.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:34:12.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836513 2024-09-10T17:34:12.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:12.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-10T17:34:12.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:12.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:34:12.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-10T17:34:12.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:12.947 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672983 2024-09-10T17:34:12.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-10T17:34:12.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-10T17:34:12.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:34:13.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-10T17:34:13.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:13.368 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-10T17:34:13.368 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:13.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:34:13.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-10T17:34:13.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:13.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-10T17:34:13.370 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509458 2024-09-10T17:34:13.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-10T17:34:13.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:34:13.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509458 2024-09-10T17:34:13.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:13.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345930 2024-09-10T17:34:13.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:13.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:34:13.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345930 2024-09-10T17:34:13.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:13.797 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 85899345930 2024-09-10T17:34:13.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345930 2024-09-10T17:34:13.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-09-10T17:34:13.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:34:14.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345930 -lt 85899345930 2024-09-10T17:34:14.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:34:14.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:14.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:14.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-10T17:34:14.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:34:14.736 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:34:14.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:34:14.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:34:14.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:34:14.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:34:14.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:34:15.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-10T17:34:15.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:34:15.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:15.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:15.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-10T17:34:15.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:34:15.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:34:15.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-10T17:34:16.025 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:16.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-09-10T17:34:16.400 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:16.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-09-10T17:34:16.778 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:16.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-09-10T17:34:17.152 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:17.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-10T17:34:17.424 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:17.424 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-10T17:34:17.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-10T17:34:17.691 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:17.691 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-10T17:34:17.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-10T17:34:17.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-09-10T17:34:18.070 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:18.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-09-10T17:34:18.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-09-10T17:34:18.450 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:18.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-09-10T17:34:19.249 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 min_size to 2 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:34:19.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:34:19.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:34:19.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:34:19.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:34:19.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:34:19.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:34:19.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:34:19.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:34:19.856 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:34:19.856 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-10T17:34:19.856 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2024-09-10T17:34:19.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:34:19.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:19.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:34:20.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836517 2024-09-10T17:34:20.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836517 2024-09-10T17:34:20.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517' 2024-09-10T17:34:20.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:20.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:34:20.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-10T17:34:20.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-10T17:34:20.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672987' 2024-09-10T17:34:20.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:20.188 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:34:20.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-10T17:34:20.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-10T17:34:20.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672987 2-64424509462' 2024-09-10T17:34:20.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:20.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:34:20.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345933 2024-09-10T17:34:20.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345933 2024-09-10T17:34:20.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672987 2-64424509462 3-85899345933' 2024-09-10T17:34:20.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:20.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836517 2024-09-10T17:34:20.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:20.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:34:20.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836517 2024-09-10T17:34:20.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:20.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836517 2024-09-10T17:34:20.435 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836517 2024-09-10T17:34:20.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-10T17:34:20.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:34:20.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836517 -lt 21474836517 2024-09-10T17:34:20.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:20.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-10T17:34:20.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:20.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:34:20.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-10T17:34:20.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:20.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-10T17:34:20.934 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672987 2024-09-10T17:34:20.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-10T17:34:20.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:34:21.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-10T17:34:21.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:21.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-10T17:34:21.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:21.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:34:21.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-10T17:34:21.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:21.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-10T17:34:21.314 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509462 2024-09-10T17:34:21.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-10T17:34:21.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:34:21.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509462 2024-09-10T17:34:21.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:21.700 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345933 2024-09-10T17:34:21.700 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:21.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:34:21.701 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345933 2024-09-10T17:34:21.701 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:21.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345933 2024-09-10T17:34:21.702 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 85899345933 2024-09-10T17:34:21.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-09-10T17:34:21.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:34:22.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345933 -lt 85899345933 2024-09-10T17:34:22.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:34:22.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:22.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:22.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-10T17:34:22.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:34:22.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:34:22.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:34:22.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:34:22.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:34:22.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:34:22.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:34:22.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-10T17:34:23.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:34:23.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:23.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-10T17:34:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:34:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:34:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-10T17:34:23.922 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:23.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-10T17:34:24.299 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:24.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-10T17:34:24.572 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:24.572 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-10T17:34:24.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-10T17:34:24.840 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:24.841 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-10T17:34:24.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-10T17:34:24.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-09-10T17:34:25.225 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:25.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-09-10T17:34:25.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-09-10T17:34:25.603 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:25.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-09-10T17:34:25.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-09-10T17:34:25.981 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:25.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-09-10T17:34:25.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-09-10T17:34:26.356 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:26.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-09-10T17:34:26.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:26.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:26.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:26.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:26.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:26.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:26.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-09-10T17:34:27.309 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-10T17:34:27.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-09-10T17:34:27.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1690: wait_for_peered: local cmd= 2024-09-10T17:34:27.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1691: wait_for_peered: local num_peered=-1 2024-09-10T17:34:27.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1692: wait_for_peered: local cur_peered 2024-09-10T17:34:27.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1693: wait_for_peered: get_timeout_delays 90 .1 2024-09-10T17:34:27.322 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:34:27.322 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:34:27.322 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:34:27.322 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:34:27.322 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:34:27.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1693: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:34:27.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1693: wait_for_peered: local -a delays 2024-09-10T17:34:27.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1694: wait_for_peered: local -i loop=0 2024-09-10T17:34:27.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1696: wait_for_peered: flush_pg_stats 2024-09-10T17:34:27.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:34:27.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:34:27.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:34:27.967 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:34:27.967 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-10T17:34:27.967 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2024-09-10T17:34:27.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:34:27.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:27.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:34:28.043 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T17:34:28.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T17:34:28.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T17:34:28.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T17:34:28.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:28.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:34:28.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672990 2024-09-10T17:34:28.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672990 2024-09-10T17:34:28.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672990' 2024-09-10T17:34:28.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:28.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:34:28.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509466 2024-09-10T17:34:28.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509466 2024-09-10T17:34:28.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672990 2-64424509466' 2024-09-10T17:34:28.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:28.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:34:28.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345937 2024-09-10T17:34:28.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345937 2024-09-10T17:34:28.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672990 2-64424509466 3-85899345937' 2024-09-10T17:34:28.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:28.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672990 2024-09-10T17:34:28.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:28.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:34:28.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672990 2024-09-10T17:34:28.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:28.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672990 2024-09-10T17:34:28.379 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672990 2024-09-10T17:34:28.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-09-10T17:34:28.380 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:34:28.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672990 2024-09-10T17:34:28.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:28.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509466 2024-09-10T17:34:28.839 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:28.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:34:28.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509466 2024-09-10T17:34:28.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:28.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509466 2024-09-10T17:34:28.841 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509466 2024-09-10T17:34:28.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-10T17:34:28.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:34:29.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-10T17:34:29.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:29.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345937 2024-09-10T17:34:29.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:29.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:34:29.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345937 2024-09-10T17:34:29.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:29.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345937 2024-09-10T17:34:29.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2024-09-10T17:34:29.349 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 85899345937 2024-09-10T17:34:29.349 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:34:29.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345937 -lt 85899345937 2024-09-10T17:34:29.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: wait_for_peered: get_num_pgs 2024-09-10T17:34:29.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:29.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:30.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: wait_for_peered: test 32 == 0 2024-09-10T17:34:30.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: true 2024-09-10T17:34:30.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: get_num_active_or_peered 2024-09-10T17:34:30.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2024-09-10T17:34:30.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-10T17:34:30.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-10T17:34:30.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-10T17:34:30.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-10T17:34:30.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: cur_peered=22 2024-09-10T17:34:30.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-10T17:34:30.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:30.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:31.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 22 = 32 2024-09-10T17:34:31.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1707: wait_for_peered: test 22 '!=' -1 2024-09-10T17:34:31.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_peered: loop=0 2024-09-10T17:34:31.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_peered: num_peered=22 2024-09-10T17:34:31.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_peered: eval 2024-09-10T17:34:31.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_peered: sleep 0.1 2024-09-10T17:34:31.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_peered: loop+=1 2024-09-10T17:34:31.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: true 2024-09-10T17:34:31.282 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: get_num_active_or_peered 2024-09-10T17:34:31.282 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2024-09-10T17:34:31.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-10T17:34:31.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-10T17:34:31.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-10T17:34:31.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-10T17:34:31.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: cur_peered=22 2024-09-10T17:34:31.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-10T17:34:31.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:31.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:32.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 22 = 32 2024-09-10T17:34:32.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1707: wait_for_peered: test 22 '!=' 22 2024-09-10T17:34:32.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: get_is_making_recovery_progress 2024-09-10T17:34:32.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T17:34:32.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T17:34:32.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T17:34:32.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T17:34:32.172 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T17:34:32.172 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T17:34:32.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T17:34:32.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T17:34:32.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_peered: (( 1 >= 13 )) 2024-09-10T17:34:32.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_peered: eval 2024-09-10T17:34:32.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_peered: sleep 0.2 2024-09-10T17:34:32.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_peered: loop+=1 2024-09-10T17:34:32.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: true 2024-09-10T17:34:32.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: get_num_active_or_peered 2024-09-10T17:34:32.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2024-09-10T17:34:32.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-10T17:34:32.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-10T17:34:32.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-10T17:34:32.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-10T17:34:33.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: cur_peered=32 2024-09-10T17:34:33.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-10T17:34:33.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:34:33.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:34:33.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 = 32 2024-09-10T17:34:33.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: break 2024-09-10T17:34:33.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1721: wait_for_peered: return 0 2024-09-10T17:34:33.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-10T17:34:34.315 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-09-10T17:34:34.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-10T17:34:34.700 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:34.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-10T17:34:34.970 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:34.970 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-10T17:34:34.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-10T17:34:35.231 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-10T17:34:35.232 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-10T17:34:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:35.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:35.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:35.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:35.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:35.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:35.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:35.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:35.351 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:34:35.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:34:35.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:34:35.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:34:35.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:34:35.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:34:35.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:35.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:34:35.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:34:35.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:35.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:34:35.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:34:35.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:34:35.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:34:35.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:35.380 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:35.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:35.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:35.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:34:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:34:35.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:34:35.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:34:35.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:34:35.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:34:35.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:35.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:34:35.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:34:35.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:35.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:34:35.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:34:35.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:34:35.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:34:35.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:35.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:35.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:34:35.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:34:35.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:34:35.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-10T17:34:35.391 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:34:35.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:35.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:35.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53672 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-10T17:34:35.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:34:35.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:34:35.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:34:35.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:34:35.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-10T17:34:35.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-10T17:34:35.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:34:35.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:34:35.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:34:35.852 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:34:35.852 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:35.853 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:35.853 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:34:35.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-10T17:34:36.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:34:36.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:34:36.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:36.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:34:36.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:34:36.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:34:36.046 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:34:36.047 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:34:36.047 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:34:36.047 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:36.047 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:36.047 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:36.047 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:34:36.047 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:36.047 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get fsid 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:34:36.127 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:36.128 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:36.128 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:36.128 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:34:36.128 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:36.128 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get mon_host 2024-09-10T17:34:36.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-09-10T17:34:36.989 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: dne 2024-09-10T17:34:36.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-09-10T17:34:37.169 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a 2024-09-10T17:34:37.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-09-10T17:34:37.312 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-10T17:34:37.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-09-10T17:34:37.454 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-10T17:34:37.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-09-10T17:34:37.653 INFO:tasks.workunit.client.0.smithi155.stderr:mon.dne does not exist 2024-09-10T17:34:37.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:37.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:37.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:37.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:37.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:37.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:37.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:37.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-10T17:34:37.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:34:37.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:34:37.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:37.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:37.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:37.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:37.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:37.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:37.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:37.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:34:37.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:34:37.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:34:37.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:34:37.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:34:37.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:34:37.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:37.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:34:37.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:34:37.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:37.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:34:37.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:34:37.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:34:37.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:34:37.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:37.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:37.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:34:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:34:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:34:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-10T17:34:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-10T17:34:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-10T17:34:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-10T17:34:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:34:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:34:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:37.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:37.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:37.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:34:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:34:37.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:34:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:34:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:34:37.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:34:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:37.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:34:37.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:34:37.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:37.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:34:37.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:34:37.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:34:37.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:34:37.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:37.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:37.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:34:37.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:34:37.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:34:37.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-10T17:34:37.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:34:37.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:37.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:37.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53672 2024-09-10T17:34:37.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:34:37.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-10T17:34:37.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-10T17:34:38.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:34:38.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:34:38.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:34:38.147 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:34:38.147 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:38.147 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:38.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:34:38.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-10T17:34:38.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:34:38.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:34:38.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:38.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:34:38.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:34:38.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:34:38.329 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:34:38.329 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:34:38.330 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:34:38.330 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:38.330 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:38.330 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:38.330 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:34:38.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:38.331 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get fsid 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:34:38.408 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:38.409 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:38.409 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:38.409 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:34:38.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:38.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get mon_host 2024-09-10T17:34:38.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-10T17:34:38.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:34:38.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:34:38.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-10T17:34:38.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:34:38.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2024-09-10T17:34:38.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-10T17:34:38.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:34:38.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:34:38.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:34:38.948 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:34:38.948 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:38.948 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:38.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:34:38.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-10T17:34:39.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:34:39.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-10T17:34:39.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:39.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-10T17:34:39.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:34:39.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:34:39.127 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-10T17:34:39.127 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-10T17:34:39.128 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-10T17:34:39.128 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:39.128 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:39.128 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:39.128 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.b.asok 2024-09-10T17:34:39.128 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:39.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.b.asok config get fsid 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-10T17:34:39.205 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:39.206 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:39.206 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:39.206 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.b.asok 2024-09-10T17:34:39.206 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:39.206 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.b.asok config get mon_host 2024-09-10T17:34:39.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-09-10T17:34:42.471 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: dne 2024-09-10T17:34:42.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-09-10T17:34:42.724 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons a 2024-09-10T17:34:42.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-09-10T17:34:42.937 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons b 2024-09-10T17:34:42.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-09-10T17:34:43.130 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a b 2024-09-10T17:34:43.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-09-10T17:34:43.380 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-10T17:34:43.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-09-10T17:34:43.683 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-10T17:34:43.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-09-10T17:34:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-10T17:34:43.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-09-10T17:34:44.160 INFO:tasks.workunit.client.0.smithi155.stderr:mon.dne does not exist 2024-09-10T17:34:44.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:44.171 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:44.171 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:44.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:44.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:44.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:44.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:44.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-10T17:34:44.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:34:44.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:34:44.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:44.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:44.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:44.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:44.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:44.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:44.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:44.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:34:44.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:34:44.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:34:44.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:34:44.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:34:44.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:34:44.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:44.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:34:44.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:34:44.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:44.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:34:44.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:34:44.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:34:44.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:34:44.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:44.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:44.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:34:44.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:34:44.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:34:44.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-10T17:34:44.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-10T17:34:44.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-10T17:34:44.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-10T17:34:44.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:34:44.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:34:44.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:34:44.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:44.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:44.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:44.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:44.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:44.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:44.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:34:44.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:34:44.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:34:44.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:34:44.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:34:44.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:34:44.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:44.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:34:44.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:34:44.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:34:44.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:34:44.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:34:44.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:34:44.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:34:44.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:44.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:44.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:34:44.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:34:44.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:34:44.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-10T17:34:44.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:34:44.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:44.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:44.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53672 2024-09-10T17:34:44.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:34:44.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:34:44.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-10T17:34:44.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-10T17:34:44.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:34:44.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:34:44.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:34:44.727 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:34:44.727 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:44.727 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:44.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:34:44.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-10T17:34:44.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:44.887 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:44.888 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:44.888 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:34:44.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:44.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get fsid 2024-09-10T17:34:44.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:34:44.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:44.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:34:44.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:34:44.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:34:44.966 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:34:44.966 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:34:44.967 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:34:44.967 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:44.967 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:44.967 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:44.967 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:34:44.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:44.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get mon_host 2024-09-10T17:34:45.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-10T17:34:45.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:34:45.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:34:45.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-10T17:34:45.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:34:45.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2024-09-10T17:34:45.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-10T17:34:45.480 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:34:45.480 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:34:45.480 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:34:45.480 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:34:45.480 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:45.480 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:45.481 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:34:45.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-10T17:34:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:45.676 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:45.676 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.b.asok 2024-09-10T17:34:45.676 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:45.676 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.b.asok config get fsid 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.b.asok 2024-09-10T17:34:45.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:45.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.b.asok config get mon_host 2024-09-10T17:34:45.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-10T17:34:45.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:34:45.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:34:45.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2024-09-10T17:34:45.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:34:45.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2024-09-10T17:34:45.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-10T17:34:46.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:34:46.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:34:46.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:34:46.292 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:34:46.292 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:46.293 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:46.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:34:46.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-10T17:34:46.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:46.488 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:46.489 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.c.asok 2024-09-10T17:34:46.489 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:46.489 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.c.asok config get fsid 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:46.570 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:46.571 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:34:46.571 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.c.asok 2024-09-10T17:34:46.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:34:46.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.c.asok config get mon_host 2024-09-10T17:34:46.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-09-10T17:34:46.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-10T17:34:46.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2024-09-10T17:34:46.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-10T17:34:46.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2024-09-10T17:34:46.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-10T17:34:46.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-10T17:34:46.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725989746 2024-09-10T17:34:46.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:46.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989686 -lt 1725989746 ]] 2024-09-10T17:34:46.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:34:46.649 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:49.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:49.845 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:49.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:49.846 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:34:49.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-10T17:34:49.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-10T17:34:49.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-10T17:34:49.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-10T17:34:49.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-10T17:34:49.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-09-10T17:34:50.140 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: dne 2024-09-10T17:34:50.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-10T17:34:50.400 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c) after stopping a 2024-09-10T17:34:50.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-10T17:34:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,c) after stopping b 2024-09-10T17:34:50.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-10T17:34:50.945 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b) after stopping c 2024-09-10T17:34:50.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-09-10T17:34:51.158 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons a b 2024-09-10T17:34:51.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-09-10T17:34:51.418 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons b c 2024-09-10T17:34:51.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-09-10T17:34:51.619 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a b c 2024-09-10T17:34:51.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-10T17:34:51.878 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-10T17:34:51.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-10T17:34:52.179 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-10T17:34:52.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-10T17:34:52.488 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-10T17:34:52.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-10T17:34:52.780 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-10T17:34:52.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-09-10T17:34:52.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:34:52.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:34:52.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:34:52.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:34:52.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:34:52.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:34:52.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-09-10T17:34:52.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-10T17:34:52.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2024-09-10T17:34:52.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-10T17:34:52.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2024-09-10T17:34:52.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-10T17:34:52.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-10T17:34:52.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725989752 2024-09-10T17:34:52.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:52.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989692 -lt 1725989752 ]] 2024-09-10T17:34:52.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:52.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:53.086 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:53.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:53.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:53.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:53.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989693 -lt 1725989752 ]] 2024-09-10T17:34:53.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:53.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:53.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:53.277 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:53.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:53.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:53.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:53.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989693 -lt 1725989752 ]] 2024-09-10T17:34:53.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:53.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:53.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:53.470 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.471 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.471 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:53.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:53.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:53.473 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:53.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989693 -lt 1725989752 ]] 2024-09-10T17:34:53.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:53.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:53.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:53.664 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:53.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:53.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:53.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:53.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989693 -lt 1725989752 ]] 2024-09-10T17:34:53.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:53.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:53.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:53.856 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:53.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:53.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:53.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:53.859 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:53.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989693 -lt 1725989752 ]] 2024-09-10T17:34:53.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:53.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:54.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:54.047 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:54.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:54.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:54.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:54.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:54.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:54.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989694 -lt 1725989752 ]] 2024-09-10T17:34:54.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:54.051 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:54.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:54.240 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:54.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:54.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:54.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:54.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:54.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:54.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989694 -lt 1725989752 ]] 2024-09-10T17:34:54.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:54.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:57.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:57.440 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:57.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:57.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:57.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:57.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:57.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:57.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989697 -lt 1725989752 ]] 2024-09-10T17:34:57.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:57.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:34:57.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:34:57.635 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:57.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:34:57.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:34:57.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:34:57.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:34:57.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:34:57.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989697 -lt 1725989752 ]] 2024-09-10T17:34:57.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:34:57.639 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:00.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:00.834 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:00.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:00.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:35:00.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:35:00.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:35:00.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:00.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989700 -lt 1725989752 ]] 2024-09-10T17:35:00.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:35:00.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:01.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:01.029 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:01.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:35:01.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:01.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:35:01.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:35:01.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:01.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989701 -lt 1725989752 ]] 2024-09-10T17:35:01.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:35:01.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:01.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:01.222 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:01.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:01.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:35:01.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:35:01.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:35:01.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:01.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989701 -lt 1725989752 ]] 2024-09-10T17:35:01.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:35:01.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:01.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:01.415 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:01.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:35:01.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:01.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:35:01.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:35:01.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:01.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989701 -lt 1725989752 ]] 2024-09-10T17:35:01.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-10T17:35:01.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:08.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:08.253 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:08.254 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:34:44.320619Z","created":"2024-09-10T17:34:44.320619Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:08.254 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-10T17:35:08.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-10T17:35:08.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-10T17:35:08.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-10T17:35:08.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-10T17:35:08.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-10T17:35:08.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-10T17:35:08.455 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons a 2024-09-10T17:35:08.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-10T17:35:08.755 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,c) after stopping b 2024-09-10T17:35:08.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-10T17:35:08.981 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons c 2024-09-10T17:35:08.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-10T17:35:09.182 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-10T17:35:09.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-10T17:35:09.400 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-10T17:35:09.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-10T17:35:09.617 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-10T17:35:09.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-10T17:35:09.819 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-10T17:35:09.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-10T17:35:09.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:09.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:09.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:09.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:09.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:09.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:09.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-10T17:35:09.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:35:09.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:35:09.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:35:09.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:09.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:09.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:09.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:09.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:09.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:09.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:35:09.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:35:09.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:35:09.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:35:09.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:35:09.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:35:09.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:35:09.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:35:09.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:35:09.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:35:09.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:35:09.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:35:09.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:35:09.941 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:35:09.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:09.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:09.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:35:09.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:35:09.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:09.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:09.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:09.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:35:09.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:35:09.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:35:09.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:35:09.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:35:09.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:35:09.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:35:09.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:35:09.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:35:09.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:35:09.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:35:09.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:35:09.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:35:09.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:35:09.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:09.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:09.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:35:09.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:35:09.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:35:09.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-10T17:35:09.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:35:09.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:09.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:09.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53672 2024-09-10T17:35:09.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:35:09.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:35:09.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:35:09.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-10T17:35:09.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-09-10T17:35:09.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-09-10T17:35:09.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-09-10T17:35:09.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-10T17:35:09.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:09.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:09.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:35:09.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:09.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-10T17:35:09.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-10T17:35:10.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:10.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:10.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:10.418 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:10.418 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:10.418 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:10.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:10.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-10T17:35:10.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:10.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:35:10.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:10.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get fsid 2024-09-10T17:35:10.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:10.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get mon_host 2024-09-10T17:35:10.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-10T17:35:10.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:10.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:10.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-10T17:35:10.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:10.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2024-09-10T17:35:10.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-10T17:35:11.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:11.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:11.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:11.240 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:11.240 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:11.240 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:11.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:11.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-10T17:35:11.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:11.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-10T17:35:11.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:11.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-10T17:35:11.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:11.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:11.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-10T17:35:11.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-10T17:35:11.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-10T17:35:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.b.asok 2024-09-10T17:35:11.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:11.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.b.asok config get fsid 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-10T17:35:11.497 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:11.498 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:11.498 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:11.498 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.b.asok 2024-09-10T17:35:11.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:11.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.b.asok config get mon_host 2024-09-10T17:35:11.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-10T17:35:11.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:11.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:11.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2024-09-10T17:35:11.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:11.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2024-09-10T17:35:11.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-10T17:35:12.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:12.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:12.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:12.069 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:12.069 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:12.069 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:12.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:12.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-10T17:35:12.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:12.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2024-09-10T17:35:12.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:12.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-10T17:35:12.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:12.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:12.262 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-10T17:35:12.262 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-10T17:35:12.263 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-10T17:35:12.263 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:12.263 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:12.263 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:12.264 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.c.asok 2024-09-10T17:35:12.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:12.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.c.asok config get fsid 2024-09-10T17:35:12.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2024-09-10T17:35:12.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:12.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-10T17:35:12.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:12.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:12.342 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-10T17:35:12.342 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-10T17:35:12.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-10T17:35:12.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:12.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:12.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:12.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.c.asok 2024-09-10T17:35:12.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:12.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.c.asok config get mon_host 2024-09-10T17:35:12.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-10T17:35:12.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:12.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:12.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2024-09-10T17:35:12.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:12.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2024-09-10T17:35:12.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-10T17:35:12.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:12.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:12.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:12.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:12.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:12.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:12.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:12.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-10T17:35:13.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:13.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2024-09-10T17:35:13.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:13.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-10T17:35:13.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:13.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:13.042 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-10T17:35:13.042 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-10T17:35:13.043 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-10T17:35:13.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:13.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:13.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:13.043 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.d.asok 2024-09-10T17:35:13.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:13.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.d.asok config get fsid 2024-09-10T17:35:13.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2024-09-10T17:35:13.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:13.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-10T17:35:13.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:13.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:13.121 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-10T17:35:13.121 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-10T17:35:13.122 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-10T17:35:13.122 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:13.122 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:13.122 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:13.122 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.d.asok 2024-09-10T17:35:13.122 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:13.122 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.d.asok config get mon_host 2024-09-10T17:35:13.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-09-10T17:35:13.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-10T17:35:13.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2024-09-10T17:35:13.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-10T17:35:13.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2024-09-10T17:35:13.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-10T17:35:13.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-10T17:35:13.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725989773 2024-09-10T17:35:13.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:13.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989713 -lt 1725989773 ]] 2024-09-10T17:35:13.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:35:13.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:26.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:26.005 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:26.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:26.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:35:26.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-10T17:35:26.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-10T17:35:26.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-10T17:35:26.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-10T17:35:26.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-10T17:35:26.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-09-10T17:35:26.276 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: dne 2024-09-10T17:35:26.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-10T17:35:26.560 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping a 2024-09-10T17:35:26.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-10T17:35:29.757 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,c,d) after stopping b 2024-09-10T17:35:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-10T17:35:30.030 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,d) after stopping c 2024-09-10T17:35:30.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-10T17:35:30.341 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,c) after stopping d 2024-09-10T17:35:30.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-09-10T17:35:30.669 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons a b 2024-09-10T17:35:30.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-09-10T17:35:30.912 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons c d 2024-09-10T17:35:30.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-10T17:35:31.263 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-10T17:35:31.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-10T17:35:31.637 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-10T17:35:31.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-10T17:35:34.833 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-10T17:35:34.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-10T17:35:38.032 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-10T17:35:38.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-10T17:35:38.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:38.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:38.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:38.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:38.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:38.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:38.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-09-10T17:35:38.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-10T17:35:38.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2024-09-10T17:35:38.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-10T17:35:38.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2024-09-10T17:35:38.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-10T17:35:38.148 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-10T17:35:38.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725989798 2024-09-10T17:35:38.149 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:38.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989738 -lt 1725989798 ]] 2024-09-10T17:35:38.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:35:38.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:38.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:38.368 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:38.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:38.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:35:38.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:35:38.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:35:38.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:38.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989738 -lt 1725989798 ]] 2024-09-10T17:35:38.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:35:38.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:41.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:41.569 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:41.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:41.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:35:41.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:35:41.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:35:41.572 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:41.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989741 -lt 1725989798 ]] 2024-09-10T17:35:41.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:35:41.573 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:44.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:44.770 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:44.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:35:44.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:44.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:35:44.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:35:44.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:44.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989744 -lt 1725989798 ]] 2024-09-10T17:35:44.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:35:44.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:44.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:44.964 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:44.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:44.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:35:44.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:35:44.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:35:44.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:44.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989744 -lt 1725989798 ]] 2024-09-10T17:35:44.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:35:44.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:35:50.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:35:50.310 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:50.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:09.975417Z","created":"2024-09-10T17:35:09.975417Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:35:50.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:35:50.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-10T17:35:50.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-10T17:35:50.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-10T17:35:50.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-10T17:35:50.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-10T17:35:50.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-10T17:35:50.646 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping a 2024-09-10T17:35:50.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-10T17:35:50.948 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons b 2024-09-10T17:35:50.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-10T17:35:51.208 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons c 2024-09-10T17:35:51.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-10T17:35:51.451 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons d 2024-09-10T17:35:51.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-10T17:35:54.639 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-10T17:35:54.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-10T17:35:54.895 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-10T17:35:54.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-10T17:35:55.155 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-10T17:35:55.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-10T17:35:55.389 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-10T17:35:55.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-09-10T17:35:55.688 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.d 2024-09-10T17:35:55.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-10T17:35:55.698 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:55.699 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:55.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:55.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:55.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:55.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:55.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:55.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:35:55.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:35:55.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:35:55.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:35:55.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:35:55.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:35:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:35:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:35:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:35:55.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:35:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:35:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:35:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:35:55.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:35:55.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:55.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:55.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:35:55.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:35:55.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:35:55.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:55.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:55.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:55.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:35:55.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:35:55.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:35:55.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:35:55.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:35:55.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:35:55.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:35:55.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:35:55.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:35:55.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:35:55.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:35:55.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:35:55.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:35:55.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:35:55.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:55.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:55.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:35:55.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:35:55.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:35:55.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-10T17:35:55.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:35:55.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:55.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:55.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53672 2024-09-10T17:35:55.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:35:55.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:35:55.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:35:55.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-10T17:35:55.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-09-10T17:35:55.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-09-10T17:35:55.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=cbd84941-ad40-4011-bbfa-a3e06f19a793 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-09-10T17:35:55.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-10T17:35:55.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:55.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:55.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:35:55.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:55.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-10T17:35:55.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-10T17:35:56.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:56.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:56.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:56.267 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:56.268 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:56.268 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:56.268 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:56.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-10T17:35:56.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:56.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:35:56.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:56.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get fsid 2024-09-10T17:35:56.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:56.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.a.asok 2024-09-10T17:35:56.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:56.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.a.asok config get mon_host 2024-09-10T17:35:56.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-10T17:35:56.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:56.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:56.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-10T17:35:56.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:56.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2024-09-10T17:35:56.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-10T17:35:57.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:57.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:57.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:57.087 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:57.087 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:57.087 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:57.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:57.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-10T17:35:57.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:57.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.b.asok 2024-09-10T17:35:57.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:57.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.b.asok config get fsid 2024-09-10T17:35:57.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.b.asok 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:57.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.b.asok config get mon_host 2024-09-10T17:35:57.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-10T17:35:57.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:57.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:57.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2024-09-10T17:35:57.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:57.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2024-09-10T17:35:57.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-10T17:35:57.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:57.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:57.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:57.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-10T17:35:58.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:58.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2024-09-10T17:35:58.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.c.asok 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:58.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.c.asok config get fsid 2024-09-10T17:35:58.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2024-09-10T17:35:58.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:58.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-10T17:35:58.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:58.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:58.174 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-10T17:35:58.174 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-10T17:35:58.175 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-10T17:35:58.175 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:58.175 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:58.175 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:58.175 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.c.asok 2024-09-10T17:35:58.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:58.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.c.asok config get mon_host 2024-09-10T17:35:58.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-10T17:35:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2024-09-10T17:35:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2024-09-10T17:35:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-10T17:35:58.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:58.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:58.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:58.721 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:58.721 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:58.721 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:58.721 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:58.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-10T17:35:58.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:58.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2024-09-10T17:35:58.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:58.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-10T17:35:58.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.d.asok 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:58.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.d.asok config get fsid 2024-09-10T17:35:58.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2024-09-10T17:35:58.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:58.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-10T17:35:58.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:58.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.d.asok 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:58.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.d.asok config get mon_host 2024-09-10T17:35:59.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-09-10T17:35:59.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-10T17:35:59.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:35:59.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2024-09-10T17:35:59.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:35:59.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/e 2024-09-10T17:35:59.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-09-10T17:35:59.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:35:59.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:59.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:59.474 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:59.474 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:59.474 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:59.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53672/$cluster-$name.asok' 2024-09-10T17:35:59.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53672/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-09-10T17:35:59.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:35:59.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2024-09-10T17:35:59.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:59.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2024-09-10T17:35:59.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:35:59.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.e.asok 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:59.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.e.asok config get fsid 2024-09-10T17:35:59.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2024-09-10T17:35:59.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:35:59.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2024-09-10T17:35:59.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:35:59.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:35:59.807 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2024-09-10T17:35:59.807 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2024-09-10T17:35:59.807 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2024-09-10T17:35:59.808 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:35:59.808 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:59.808 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:35:59.808 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53672/ceph-mon.e.asok 2024-09-10T17:35:59.808 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:35:59.808 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53672/ceph-mon.e.asok config get mon_host 2024-09-10T17:35:59.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-09-10T17:35:59.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-10T17:35:59.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2024-09-10T17:35:59.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-10T17:35:59.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2024-09-10T17:35:59.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-10T17:35:59.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-10T17:35:59.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725989819 2024-09-10T17:35:59.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:35:59.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989759 -lt 1725989819 ]] 2024-09-10T17:35:59.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-09-10T17:35:59.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:04.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:04.600 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:04.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2024-09-10T17:36:04.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:04.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-10T17:36:04.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-10T17:36:04.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-10T17:36:04.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-10T17:36:04.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-10T17:36:04.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-09-10T17:36:04.879 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: dne 2024-09-10T17:36:04.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-10T17:36:08.075 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d,e) after stopping a 2024-09-10T17:36:08.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-10T17:36:08.383 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,c,d,e) after stopping b 2024-09-10T17:36:08.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-10T17:36:08.660 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,d,e) after stopping c 2024-09-10T17:36:08.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-10T17:36:09.001 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,c,e) after stopping d 2024-09-10T17:36:09.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-10T17:36:09.281 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,c,d) after stopping e 2024-09-10T17:36:09.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-09-10T17:36:12.479 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (c,d,e) after stopping a b 2024-09-10T17:36:12.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-09-10T17:36:12.761 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,e) after stopping c d 2024-09-10T17:36:12.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-09-10T17:36:13.005 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons a b c 2024-09-10T17:36:13.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-10T17:36:13.297 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-10T17:36:13.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-10T17:36:13.673 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-10T17:36:13.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-10T17:36:13.952 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-10T17:36:13.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-10T17:36:14.237 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-10T17:36:14.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-10T17:36:14.541 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.d 2024-09-10T17:36:14.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-10T17:36:14.816 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.e 2024-09-10T17:36:14.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-10T17:36:14.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:36:14.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:36:14.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:36:14.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:36:14.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:36:14.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:36:14.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-09-10T17:36:14.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-10T17:36:14.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2024-09-10T17:36:14.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-10T17:36:14.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2024-09-10T17:36:14.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-10T17:36:14.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-10T17:36:14.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725989834 2024-09-10T17:36:14.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:14.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989774 -lt 1725989834 ]] 2024-09-10T17:36:14.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:14.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:15.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:15.123 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:15.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:15.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:15.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:15.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989775 -lt 1725989834 ]] 2024-09-10T17:36:15.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:15.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:15.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:15.316 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:15.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:15.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:15.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:15.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989775 -lt 1725989834 ]] 2024-09-10T17:36:15.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:15.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:15.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:15.510 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:15.510 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:15.510 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:15.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:15.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:15.513 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:15.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989775 -lt 1725989834 ]] 2024-09-10T17:36:15.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:15.515 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:15.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:15.703 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:15.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:15.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:15.706 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989775 -lt 1725989834 ]] 2024-09-10T17:36:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:15.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:18.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:18.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:18.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:18.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:18.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:18.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989778 -lt 1725989834 ]] 2024-09-10T17:36:18.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:18.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:19.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:19.096 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:19.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:19.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:19.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:19.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989779 -lt 1725989834 ]] 2024-09-10T17:36:19.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:19.100 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:19.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:19.290 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:19.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:19.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:19.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:19.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989779 -lt 1725989834 ]] 2024-09-10T17:36:19.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:19.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:19.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:19.483 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:19.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:19.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:19.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:19.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989779 -lt 1725989834 ]] 2024-09-10T17:36:19.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:19.489 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:19.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:19.675 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.676 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.676 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:19.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:19.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:19.678 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:19.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989779 -lt 1725989834 ]] 2024-09-10T17:36:19.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:19.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:19.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:19.867 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:19.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:19.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:19.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:19.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:19.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989779 -lt 1725989834 ]] 2024-09-10T17:36:19.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:19.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:20.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:20.061 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:20.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:20.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:20.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:20.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989780 -lt 1725989834 ]] 2024-09-10T17:36:20.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:20.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:20.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:20.254 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.254 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.254 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:20.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:20.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989780 -lt 1725989834 ]] 2024-09-10T17:36:20.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:20.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:20.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:20.446 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:20.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:20.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:20.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:20.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989780 -lt 1725989834 ]] 2024-09-10T17:36:20.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:20.450 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:20.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:20.638 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.639 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.639 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:20.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:20.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:20.642 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:20.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989780 -lt 1725989834 ]] 2024-09-10T17:36:20.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:20.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:20.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:20.831 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:20.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:20.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:20.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:20.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:20.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989780 -lt 1725989834 ]] 2024-09-10T17:36:20.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:20.836 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:21.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:21.024 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:21.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:21.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:21.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:21.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:21.028 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:21.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989781 -lt 1725989834 ]] 2024-09-10T17:36:21.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:21.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:24.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:24.225 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:24.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:24.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:24.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:24.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:24.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:24.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989784 -lt 1725989834 ]] 2024-09-10T17:36:24.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-10T17:36:24.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:30.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:30.450 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:30.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:30.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-10T17:36:30.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-10T17:36:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-10T17:36:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-10T17:36:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-10T17:36:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-10T17:36:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-10T17:36:30.713 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d,e) after stopping a 2024-09-10T17:36:30.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-10T17:36:30.934 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (c,d,e) after stopping b 2024-09-10T17:36:30.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-10T17:36:31.207 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,d,e) after stopping c 2024-09-10T17:36:31.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-10T17:36:31.637 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,e) after stopping d 2024-09-10T17:36:31.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-10T17:36:31.938 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping e 2024-09-10T17:36:31.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-10T17:36:32.214 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-10T17:36:32.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-10T17:36:32.466 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-10T17:36:32.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-10T17:36:32.753 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-10T17:36:32.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-10T17:36:33.013 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-10T17:36:33.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-10T17:36:33.247 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.d 2024-09-10T17:36:33.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-10T17:36:36.443 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.e 2024-09-10T17:36:36.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-09-10T17:36:36.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:36:36.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:36:36.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:36:36.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:36:36.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:36:36.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:36:36.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-09-10T17:36:36.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-10T17:36:36.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2024-09-10T17:36:36.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-10T17:36:36.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2024-09-10T17:36:36.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-10T17:36:36.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-10T17:36:36.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725989856 2024-09-10T17:36:36.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:36.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989796 -lt 1725989856 ]] 2024-09-10T17:36:36.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:36.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:36.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:36.751 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:36.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:36.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:36.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:36.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:36.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:36.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989796 -lt 1725989856 ]] 2024-09-10T17:36:36.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:36.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:36.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:36.943 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:36.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:36.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:36.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:36.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:36.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:36.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989796 -lt 1725989856 ]] 2024-09-10T17:36:36.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:36.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:37.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:37.136 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:37.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:37.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:37.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:37.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:37.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:37.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989797 -lt 1725989856 ]] 2024-09-10T17:36:37.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:37.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:37.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:37.328 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:37.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:37.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:37.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:37.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:37.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:37.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989797 -lt 1725989856 ]] 2024-09-10T17:36:37.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:37.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:40.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:40.528 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:40.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:40.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:40.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:40.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:40.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:40.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989800 -lt 1725989856 ]] 2024-09-10T17:36:40.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:40.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:40.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:40.722 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:40.723 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:40.723 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:40.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:40.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:40.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:40.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989800 -lt 1725989856 ]] 2024-09-10T17:36:40.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:40.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:40.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:40.916 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:40.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:40.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:40.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:40.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:40.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:40.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989800 -lt 1725989856 ]] 2024-09-10T17:36:40.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:40.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:41.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:41.110 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.110 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:41.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:41.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:41.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:41.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989801 -lt 1725989856 ]] 2024-09-10T17:36:41.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:41.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:41.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:41.302 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:41.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:41.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:41.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:41.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989801 -lt 1725989856 ]] 2024-09-10T17:36:41.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:41.306 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:41.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:41.495 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:41.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:41.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:41.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:41.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989801 -lt 1725989856 ]] 2024-09-10T17:36:41.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:41.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:41.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:41.687 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:41.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:41.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:41.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:41.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989801 -lt 1725989856 ]] 2024-09-10T17:36:41.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:41.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:41.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:41.879 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:41.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:41.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:41.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:41.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:41.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989801 -lt 1725989856 ]] 2024-09-10T17:36:41.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:41.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:42.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:42.073 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:42.074 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:42.074 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:42.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:42.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:42.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:42.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989802 -lt 1725989856 ]] 2024-09-10T17:36:42.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:42.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:42.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:42.263 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:42.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:42.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:42.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:42.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:42.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:42.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989802 -lt 1725989856 ]] 2024-09-10T17:36:42.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:42.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:42.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:42.456 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:42.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:42.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:42.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-10T17:36:42.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-10T17:36:42.460 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-10T17:36:42.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725989802 -lt 1725989856 ]] 2024-09-10T17:36:42.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-10T17:36:42.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-10T17:36:49.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-10T17:36:49.434 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:49.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cbd84941-ad40-4011-bbfa-a3e06f19a793","modified":"2024-09-10T17:35:55.853851Z","created":"2024-09-10T17:35:55.853851Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-10T17:36:49.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-10T17:36:49.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-10T17:36:49.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-10T17:36:49.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-10T17:36:49.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-10T17:36:49.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-10T17:36:49.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-10T17:36:49.739 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping a 2024-09-10T17:36:49.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-10T17:36:49.959 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons b 2024-09-10T17:36:49.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-10T17:36:53.107 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons c 2024-09-10T17:36:53.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-10T17:36:53.395 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons d 2024-09-10T17:36:53.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-10T17:36:53.670 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping e 2024-09-10T17:36:53.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-10T17:36:53.890 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-10T17:36:53.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-10T17:36:54.174 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-10T17:36:54.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-10T17:36:54.393 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.b would break quorum 2024-09-10T17:36:54.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-10T17:36:54.653 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-10T17:36:54.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-10T17:37:00.803 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.d would break quorum 2024-09-10T17:37:00.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-10T17:37:03.994 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.e 2024-09-10T17:37:04.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-10T17:37:04.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:37:04.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:37:04.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:37:04.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:37:04.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:37:04.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:37:04.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-10T17:37:04.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:37:04.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:37:04.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:37:04.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:37:04.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:37:04.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:37:04.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:37:04.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:37:04.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:37:04.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:37:04.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:37:04.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:37:04.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:37:04.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:37:04.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:37:04.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:37:04.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:37:04.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:37:04.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:37:04.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:37:04.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:37:04.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:37:04.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:37:04.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:04.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:37:04.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:37:04.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:37:04.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:37:04.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T17:37:04.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/ok-to-stop 0 2024-09-10T17:37:04.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-10T17:37:04.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T17:37:04.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-10T17:37:04.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:37:04.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:37:04.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:37:04.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:37:04.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:37:04.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:37:04.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:37:04.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:37:04.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:37:04.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:37:04.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:37:04.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:37:04.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:37:04.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:37:04.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:37:04.141 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:37:04.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:37:04.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T17:37:04.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-10T17:37:04.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:37:04.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:04.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53672 2024-09-10T17:37:04.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53672 2024-09-10T17:37:04.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:37:04.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:37:04.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T17:37:04.147 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T17:37:04.147 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T17:37:04.212 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-09-10T17:37:04.212 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-09-10T17:37:04.274 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T17:37:04.279 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T17:37:04.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/rados-striper 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-09-10T17:37:04.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-09-10T17:37:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none ' 2024-09-10T17:37:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-09-10T17:37:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-09-10T17:37:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/rados-striper 2024-09-10T17:37:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/rados-striper 2024-09-10T17:37:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2024-09-10T17:37:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:37:04.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2024-09-10T17:37:04.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:37:04.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:37:04.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:37:04.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:37:04.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:37:04.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:37:04.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:37:04.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:37:04.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:37:04.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:37:04.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:37:04.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:37:04.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:37:04.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:37:04.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:37:04.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:37:04.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:37:04.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:37:04.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2024-09-10T17:37:04.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.70556 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/rados-striper 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.70556 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:37:04.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/rados-striper 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/rados-striper/a 2024-09-10T17:37:04.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-09-10T17:37:04.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:37:04.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:37:04.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:37:04.757 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:37:04.758 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:04.758 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:37:04.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.70556/$cluster-$name.asok' 2024-09-10T17:37:04.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:37:04.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:37:04.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:37:04.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:37:04.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:37:04.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:37:04.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:37:04.951 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:37:04.952 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:37:04.952 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:37:04.952 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:37:04.952 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:04.952 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:37:04.952 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.70556/ceph-mon.a.asok 2024-09-10T17:37:04.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:37:04.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.70556/ceph-mon.a.asok config get fsid 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:05.031 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:37:05.032 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.70556/ceph-mon.a.asok 2024-09-10T17:37:05.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:37:05.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.70556/ceph-mon.a.asok config get mon_host 2024-09-10T17:37:05.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-09-10T17:37:05.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2024-09-10T17:37:05.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:37:05.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/0 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:37:05.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.70556/$cluster-$name.asok' 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:37:05.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/0 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=06eb2139-fc13-4ca8-a6d2-556d24153e49 2024-09-10T17:37:05.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 06eb2139-fc13-4ca8-a6d2-556d24153e49' 2024-09-10T17:37:05.117 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 06eb2139-fc13-4ca8-a6d2-556d24153e49 2024-09-10T17:37:05.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:37:05.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDBg+BmU5SzBxAAwS3URilK5pIspADogvkdpQ== 2024-09-10T17:37:05.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDBg+BmU5SzBxAAwS3URilK5pIspADogvkdpQ=="}' 2024-09-10T17:37:05.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 06eb2139-fc13-4ca8-a6d2-556d24153e49 -i td/rados-striper/0/new.json 2024-09-10T17:37:05.982 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:37:05.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/0/new.json 2024-09-10T17:37:05.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBg+BmU5SzBxAAwS3URilK5pIspADogvkdpQ== --osd-uuid 06eb2139-fc13-4ca8-a6d2-556d24153e49 2024-09-10T17:37:06.019 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:06.018+0000 7fb34de1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:06.051 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:06.050+0000 7fb34de1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:06.052 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:06.051+0000 7fb34de1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:06.052 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:06.051+0000 7fb34de1a640 -1 bdev(0x563f19ca5c00 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:37:06.052 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:06.051+0000 7fb34de1a640 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-09-10T17:37:11.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/0/keyring 2024-09-10T17:37:11.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:37:11.454 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:37:11.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:37:11.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:37:11.656 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:37:11.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:37:11.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:37:11.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:37:11.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:37:11.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:37:11.682 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:11.681+0000 7f7cacb58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:11.685 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:11.685+0000 7f7cacb58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:11.686 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:11.686+0000 7f7cacb58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:11.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:37:11.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:37:11.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:37:11.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:37:11.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:37:11.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:11.888 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:37:11.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:37:11.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:11.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:12.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:12.783 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:12.783+0000 7f7cacb58640 -1 Falling back to public interface 2024-09-10T17:37:13.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:13.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:13.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:37:13.090 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:37:13.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:13.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:13.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:14.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:14.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:14.286 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:37:14.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:37:14.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:14.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:14.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:14.636 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:14.635+0000 7f7cacb58640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:37:15.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:15.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:15.483 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:37:15.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:37:15.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:15.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:15.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:16.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:16.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:16.743 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:37:16.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:37:16.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:16.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:16.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:17.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:17.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:17.937 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:37:17.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:37:17.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:17.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:18.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:19.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:19.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:19.137 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:37:19.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:37:19.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:19.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:19.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:20.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:20.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:20.337 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:37:20.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:37:20.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:20.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:20.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:21.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:21.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:21.530 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:37:21.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:37:21.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:21.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:21.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:22.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:22.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:22.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:37:22.725 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:37:22.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:22.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:22.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:23.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:23.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:23.921 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:37:23.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:37:23.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:23.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:24.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:25.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:25.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:25.117 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:37:25.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:37:25.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:25.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:25.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:26.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:26.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:26.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:37:26.309 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:37:26.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:26.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:26.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:27.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:27.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:27.506 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:37:27.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:37:27.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:27.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:27.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:28.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:28.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:28.699 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:37:28.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:37:28.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:28.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:28.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:29.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:29.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:29.894 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:37:29.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:37:29.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:29.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:30.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:31.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:31.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:31.087 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:37:31.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:37:31.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:31.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:31.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:32.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:32.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:32.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:37:32.281 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:37:32.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:32.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:32.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:33.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:33.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:33.476 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-10T17:37:33.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:37:33.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:33.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:33.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:34.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:34.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:34.670 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-10T17:37:34.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T17:37:34.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:34.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:34.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:35.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:35.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:35.867 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-10T17:37:35.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-10T17:37:35.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:35.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:36.097 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3200108155,v1:127.0.0.1:6801/3200108155] [v2:127.0.0.1:6802/3200108155,v1:127.0.0.1:6803/3200108155] exists,up 06eb2139-fc13-4ca8-a6d2-556d24153e49 2024-09-10T17:37:36.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:37:36.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:37:36.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:37:36.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-09-10T17:37:36.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/1 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-09-10T17:37:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.70556/$cluster-$name.asok' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:37:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/1 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:37:36.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2b5e82e0-5715-4263-a82a-9eca22d1eedd 2024-09-10T17:37:36.105 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 2b5e82e0-5715-4263-a82a-9eca22d1eedd 2024-09-10T17:37:36.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2b5e82e0-5715-4263-a82a-9eca22d1eedd' 2024-09-10T17:37:36.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:37:36.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDgg+Bmrzn2BhAApa0HEoD7FUlEKoM4zxnH3g== 2024-09-10T17:37:36.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDgg+Bmrzn2BhAApa0HEoD7FUlEKoM4zxnH3g=="}' 2024-09-10T17:37:36.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2b5e82e0-5715-4263-a82a-9eca22d1eedd -i td/rados-striper/1/new.json 2024-09-10T17:37:36.401 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:37:36.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/1/new.json 2024-09-10T17:37:36.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgg+Bmrzn2BhAApa0HEoD7FUlEKoM4zxnH3g== --osd-uuid 2b5e82e0-5715-4263-a82a-9eca22d1eedd 2024-09-10T17:37:36.439 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:36.437+0000 7f02299dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:36.440 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:36.439+0000 7f02299dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:36.441 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:36.440+0000 7f02299dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:36.442 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:36.441+0000 7f02299dd640 -1 bdev(0x560bf8493c00 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:37:36.442 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:36.441+0000 7f02299dd640 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-09-10T17:37:42.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/1/keyring 2024-09-10T17:37:42.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:37:42.115 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-10T17:37:42.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:37:42.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:37:42.322 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-10T17:37:42.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:37:42.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:37:42.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:37:42.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:37:42.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:37:42.347 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:42.346+0000 7f38b6c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:42.349 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:42.348+0000 7f38b6c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:42.350 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:42.349+0000 7f38b6c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:42.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:37:42.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:37:42.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:37:42.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:37:42.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:37:42.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:42.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:37:42.521 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:37:42.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:42.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:42.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:43.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:43.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:43.733 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:37:43.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:37:43.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:43.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:43.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:43.936+0000 7f38b6c1b640 -1 Falling back to public interface 2024-09-10T17:37:44.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:44.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:44.924 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:37:44.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:37:44.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:44.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:45.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:45.781 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:37:45.779+0000 7f38b6c1b640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:37:46.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:46.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:46.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:37:46.117 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:37:46.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:46.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:46.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:47.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:47.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:47.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:37:47.531 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:37:47.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:47.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:48.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:48.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:48.726 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:37:48.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:37:48.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:48.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:48.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:49.924 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:37:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:37:49.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:49.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:50.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:51.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:51.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:51.115 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:37:51.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:37:51.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:51.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:51.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:52.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:52.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:52.307 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:37:52.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:37:52.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:52.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:52.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:53.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:53.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:53.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:37:53.503 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:37:53.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:53.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:53.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:54.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:54.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:54.699 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:37:54.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:37:54.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:54.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:54.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:55.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:55.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:55.893 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:37:55.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:37:55.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:55.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:56.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:57.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:57.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:57.086 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:37:57.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:37:57.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:57.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:57.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:58.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:58.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:58.279 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:37:58.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:37:58.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:58.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:58.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:59.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:59.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:59.475 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:37:59.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:37:59.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:59.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:59.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:00.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:00.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:00.674 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:38:00.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:38:00.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:00.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:00.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:01.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:01.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:01.868 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:38:01.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:38:01.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:01.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:02.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:03.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:03.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:03.063 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:38:03.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:38:03.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:03.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:03.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:04.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:04.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:04.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:38:04.261 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-10T17:38:04.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:04.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:04.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:05.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:05.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:05.456 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-10T17:38:05.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T17:38:05.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:05.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:05.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:06.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:06.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:06.651 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-10T17:38:06.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-10T17:38:06.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:06.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:06.868 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3399768975,v1:127.0.0.1:6809/3399768975] [v2:127.0.0.1:6810/3399768975,v1:127.0.0.1:6811/3399768975] exists,up 2b5e82e0-5715-4263-a82a-9eca22d1eedd 2024-09-10T17:38:06.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/2 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:38:06.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.70556/$cluster-$name.asok' 2024-09-10T17:38:06.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:38:06.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/2 2024-09-10T17:38:06.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:38:06.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=545c2c95-0641-4739-b824-0859d9bb8633 2024-09-10T17:38:06.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 545c2c95-0641-4739-b824-0859d9bb8633' 2024-09-10T17:38:06.875 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 545c2c95-0641-4739-b824-0859d9bb8633 2024-09-10T17:38:06.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:38:06.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+g+Bmt77nNBAAsIKQUBUdfiarpFryeIvDUw== 2024-09-10T17:38:06.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+g+Bmt77nNBAAsIKQUBUdfiarpFryeIvDUw=="}' 2024-09-10T17:38:06.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 545c2c95-0641-4739-b824-0859d9bb8633 -i td/rados-striper/2/new.json 2024-09-10T17:38:07.276 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:38:07.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/2/new.json 2024-09-10T17:38:07.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+g+Bmt77nNBAAsIKQUBUdfiarpFryeIvDUw== --osd-uuid 545c2c95-0641-4739-b824-0859d9bb8633 2024-09-10T17:38:07.313 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:07.312+0000 7f737f4ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:07.370 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:07.369+0000 7f737f4ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:07.371 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:07.370+0000 7f737f4ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:07.372 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:07.371+0000 7f737f4ef640 -1 bdev(0x55c291a79c00 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:38:07.372 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:07.371+0000 7f737f4ef640 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-09-10T17:38:13.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/2/keyring 2024-09-10T17:38:13.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:38:13.303 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-10T17:38:13.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:38:13.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:38:13.508 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-10T17:38:13.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:38:13.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0eb20198-a037-4654-83e9-661cf55ae868 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70556/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:38:13.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:38:13.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:38:13.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:38:13.534 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:13.533+0000 7f74f30e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:13.536 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:13.535+0000 7f74f30e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:13.537 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:13.536+0000 7f74f30e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:13.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:38:13.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:38:13.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:38:13.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:38:13.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:38:13.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:13.699 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:38:13.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:38:13.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:13.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:13.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:14.381 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:14.380+0000 7f74f30e2640 -1 Falling back to public interface 2024-09-10T17:38:14.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:14.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:14.902 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:38:14.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:38:14.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:14.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:15.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:16.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:16.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:16.096 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:38:16.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:38:16.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:16.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:16.206 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:16.204+0000 7f74f30e2640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:38:16.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:17.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:17.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:17.313 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:38:17.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:38:17.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:17.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:17.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:18.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:18.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:18.511 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:38:18.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:38:18.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:18.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:18.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:19.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:19.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:19.706 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:38:19.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:38:19.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:19.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:19.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:20.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:20.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:20.902 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:38:20.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:38:20.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:20.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:21.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:22.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:22.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:22.098 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:38:22.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:38:22.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:22.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:22.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:23.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:23.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:23.294 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:38:23.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:38:23.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:23.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:23.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:24.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:24.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:24.491 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:38:24.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:38:24.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:24.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:24.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:25.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:25.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:25.688 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:38:25.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:38:25.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:25.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:25.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:26.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:26.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:26.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:38:26.886 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:38:26.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:26.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:27.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:28.080 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:38:28.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:38:28.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:28.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:28.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:29.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:29.271 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:38:29.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:29.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:38:29.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:29.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:29.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:30.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:30.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:30.469 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:38:30.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:38:30.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:30.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:30.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:31.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:31.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:31.664 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:38:31.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:38:31.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:31.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:31.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:32.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:32.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:32.861 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:38:32.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:38:32.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:32.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:33.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:34.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:34.051 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:38:34.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:34.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:38:34.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:34.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:34.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:35.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:35.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:35.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:38:35.248 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-10T17:38:35.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:35.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:35.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:36.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:36.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:36.441 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-10T17:38:36.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T17:38:36.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:36.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:36.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:37.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:37.636 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-10T17:38:37.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:37.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-10T17:38:37.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:37.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:37.825 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/42216817,v1:127.0.0.1:6817/42216817] [v2:127.0.0.1:6818/42216817,v1:127.0.0.1:6819/42216817] exists,up 545c2c95-0641-4739-b824-0859d9bb8633 2024-09-10T17:38:37.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:38:37.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:38:37.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:38:37.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-09-10T17:38:37.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:38:38.021 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:38:38.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:38:38.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:38:38.398 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:38:38.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:38:39.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:38:40.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-09-10T17:38:40.753 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records in 2024-09-10T17:38:40.753 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records out 2024-09-10T17:38:40.753 INFO:tasks.workunit.client.0.smithi155.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 5.8459e-05 s, 21.1 MB/s 2024-09-10T17:38:40.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-09-10T17:38:41.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-09-10T17:38:41.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-09-10T17:38:41.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-10T17:38:41.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-09-10T17:38:41.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-09-10T17:38:41.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-09-10T17:38:41.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-09-10T17:38:41.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-09-10T17:38:41.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-09-10T17:38:41.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-09-10T17:38:41.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-09-10T17:38:41.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-09-10T17:38:41.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-09-10T17:38:41.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-09-10T17:38:42.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-09-10T17:38:42.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-09-10T17:38:42.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-10T17:38:42.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-09-10T17:38:42.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-09-10T17:38:42.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-09-10T17:38:42.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-09-10T17:38:42.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-09-10T17:38:42.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-10T17:38:42.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-10T17:38:42.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-09-10T17:38:42.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-09-10T17:38:42.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-09-10T17:38:42.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-09-10T17:38:42.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-09-10T17:38:42.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-09-10T17:38:42.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-09-10T17:38:42.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-09-10T17:38:42.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-09-10T17:38:42.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-09-10T17:38:42.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-09-10T17:38:42.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-09-10T17:38:42.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-09-10T17:38:42.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-09-10T17:38:42.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/rados-striper 2024-09-10T17:38:42.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T17:38:42.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local 'expected=No data available' 2024-09-10T17:38:42.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T17:38:42.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T17:38:42.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-10T17:38:42.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T17:38:42.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T17:38:42.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-10T17:38:42.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-10T17:38:42.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-10T17:38:42.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/rados-striper 2024-09-10T17:38:42.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T17:38:42.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local 'expected=No data available' 2024-09-10T17:38:42.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T17:38:42.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T17:38:42.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-10T17:38:42.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T17:38:42.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T17:38:42.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-10T17:38:42.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-10T17:38:42.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-09-10T17:38:42.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-09-10T17:38:42.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/rados-striper 2024-09-10T17:38:42.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T17:38:42.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local 'expected=No such file or directory' 2024-09-10T17:38:42.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T17:38:42.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T17:38:42.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: rados --pool rbd --striper stat toyfile 2024-09-10T17:38:42.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T17:38:42.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T17:38:42.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-10T17:38:42.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-10T17:38:42.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-09-10T17:38:42.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/rados-striper 2024-09-10T17:38:42.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T17:38:42.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local 'expected=No such file or directory' 2024-09-10T17:38:42.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T17:38:42.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T17:38:42.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-09-10T17:38:42.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T17:38:42.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T17:38:42.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:38:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:38:42.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:38:42.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:38:42.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:38:42.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:38:43.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:38:43.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:38:43.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:38:43.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:38:43.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:38:43.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:38:43.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:38:43.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:38:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:38:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2024-09-10T17:38:43.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:38:43.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:43.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:38:43.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.70556 2024-09-10T17:38:43.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:38:43.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:38:43.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T17:38:43.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/rados-striper 0 2024-09-10T17:38:43.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2024-09-10T17:38:43.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T17:38:43.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2024-09-10T17:38:43.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:38:43.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:38:43.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:38:43.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:38:43.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:38:43.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:38:43.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:38:43.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:38:43.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:38:43.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:38:43.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:38:43.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:38:43.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:38:43.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:38:43.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:38:43.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:38:43.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:38:43.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T17:38:43.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2024-09-10T17:38:43.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:38:43.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:43.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.70556 2024-09-10T17:38:43.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.70556 2024-09-10T17:38:43.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:38:43.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:38:43.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T17:38:43.027 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T17:38:43.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T17:38:43.091 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-09-10T17:38:43.091 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-09-10T17:38:43.154 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T17:38:43.159 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T17:38:43.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: export .:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T17:38:43.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2336: export: `.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin': not a valid identifier 2024-09-10T17:38:43.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2338: run_tests: export CEPH_MON=127.0.0.1:7109 2024-09-10T17:38:43.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2338: run_tests: CEPH_MON=127.0.0.1:7109 2024-09-10T17:38:43.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: export CEPH_ARGS 2024-09-10T17:38:43.159 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: uuidgen 2024-09-10T17:38:43.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: CEPH_ARGS+=' --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none ' 2024-09-10T17:38:43.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2341: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-09-10T17:38:43.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2342: run_tests: export CEPH_CONF=/dev/null 2024-09-10T17:38:43.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2342: run_tests: CEPH_CONF=/dev/null 2024-09-10T17:38:43.160 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: run_tests: set 2024-09-10T17:38:43.160 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-09-10T17:38:43.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: run_tests: local 'funcs=test_activate_osd 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_activate_osd_after_mark_down 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_activate_osd_skip_benchmark 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_destroy_osd 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_display_logs 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_erasure_code_plugin_exists 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_expect_failure 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_flush_pg_stats 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_config 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_is_making_recovery_progress 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_last_scrub_stamp 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_not_primary 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_num_active_clean 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_num_active_or_peered 2024-09-10T17:38:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_num_pgs 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_op_scheduler 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_osd_id_used_by_pgs 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_osds 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_pg 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_primary 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_timeout_delays 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_is_clean 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_kill_daemon 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_kill_daemons 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_objectstore_tool 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_pg_schedule_scrub 2024-09-10T17:38:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:test_pg_scrub 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_repair 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_run_in_background 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_run_mon 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_run_osd 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_set_config 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_setup 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_teardown 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_background 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_clean 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_health_ok 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_osd 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_peered 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_scrub 2024-09-10T17:38:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_osd_id_used_by_pgs' 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: run_tests: local dir=td/ceph-helpers 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_activate_osd td/ceph-helpers 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: test_activate_osd: local dir=td/ceph-helpers 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: test_activate_osd: setup td/ceph-helpers 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:38:43.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:38:43.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:38:43.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:38:43.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:38:43.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:38:43.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:38:43.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:38:43.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:38:43.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:38:43.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:38:43.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:38:43.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:38:43.169 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:38:43.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:38:43.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:38:43.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:38:43.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:38:43.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:43.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:38:43.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:38:43.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:38:43.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:38:43.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:38:43.172 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:38:43.172 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:43.172 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:38:43.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: test_activate_osd: run_mon td/ceph-helpers a 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:38:43.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:38:43.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-10T17:38:43.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:38:43.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:38:43.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:38:43.634 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:38:43.634 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:43.634 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:38:43.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:38:43.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:38:43.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:38:43.836 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:38:43.836 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:38:43.836 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:38:43.836 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:38:43.836 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:38:43.836 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:38:43.836 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:38:43.837 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:38:43.837 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:38:43.837 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:43.837 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:38:43.837 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:38:43.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:38:43.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:38:43.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:38:43.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:38:43.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:38:43.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: test_activate_osd: run_mgr td/ceph-helpers x 2024-09-10T17:38:43.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:38:43.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:38:43.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:38:43.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:38:43.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:38:43.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:38:44.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:38:44.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:38:44.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:38:44.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:38:44.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:44.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:38:44.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:38:44.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:38:44.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:38:44.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:884: test_activate_osd: run_osd td/ceph-helpers 0 2024-09-10T17:38:44.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:38:44.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:38:44.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:38:44.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:38:44.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:38:44.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:38:44.872 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:38:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:38:44.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:38:44.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=29c28616-2ee3-4b70-a401-f23f8969b42a 2024-09-10T17:38:44.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 29c28616-2ee3-4b70-a401-f23f8969b42a' 2024-09-10T17:38:44.874 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 29c28616-2ee3-4b70-a401-f23f8969b42a 2024-09-10T17:38:44.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:38:44.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAkhOBmMygINRAA9bw1lmET5rLOFkJXAY28lA== 2024-09-10T17:38:44.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAkhOBmMygINRAA9bw1lmET5rLOFkJXAY28lA=="}' 2024-09-10T17:38:44.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 29c28616-2ee3-4b70-a401-f23f8969b42a -i td/ceph-helpers/0/new.json 2024-09-10T17:38:45.168 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:38:45.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:38:45.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkhOBmMygINRAA9bw1lmET5rLOFkJXAY28lA== --osd-uuid 29c28616-2ee3-4b70-a401-f23f8969b42a 2024-09-10T17:38:45.212 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:45.211+0000 7f2e3f94f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:45.237 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:45.236+0000 7f2e3f94f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:45.238 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:45.237+0000 7f2e3f94f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:45.239 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:45.238+0000 7f2e3f94f640 -1 bdev(0x55d502e1fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:38:45.239 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:45.238+0000 7f2e3f94f640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:38:51.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:38:51.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:38:51.268 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:38:51.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:38:51.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:38:51.813 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:38:51.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:38:51.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:38:51.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:38:51.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:38:51.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:38:51.841 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:51.839+0000 7f3769014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:51.843 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:51.842+0000 7f3769014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:51.844 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:51.843+0000 7f3769014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:52.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:38:52.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:38:52.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:38:52.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:38:52.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:38:52.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:52.241 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:38:52.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:38:52.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:52.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:38:52.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:53.195 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:53.194+0000 7f3769014640 -1 Falling back to public interface 2024-09-10T17:38:53.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:53.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:53.630 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:38:53.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:38:53.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:53.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:38:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:54.730 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:38:54.728+0000 7f3769014640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:38:55.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:55.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:55.069 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:38:55.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:38:55.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:55.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:38:55.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:56.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:56.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:56.473 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:38:56.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:38:56.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:56.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:38:56.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:57.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:38:57.866 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:38:57.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:57.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:38:58.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:59.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:59.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:59.293 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:38:59.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:38:59.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:59.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:38:59.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:00.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:00.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:00.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:39:00.857 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:39:00.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:00.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:01.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:02.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:02.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:02.296 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:39:02.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:39:02.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:02.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:02.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:03.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:03.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:03.810 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:39:03.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:39:03.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:03.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:04.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:05.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:05.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:05.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:39:05.224 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:39:05.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:05.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:05.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:06.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:06.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:06.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:39:06.765 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:39:06.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:06.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:07.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:08.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:08.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:08.476 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:39:08.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:39:08.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:08.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:09.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:10.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:10.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:10.140 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:39:10.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:39:10.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:10.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:10.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:11.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:11.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:11.893 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:39:11.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:39:11.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:11.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:12.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:13.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:13.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:13.977 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:39:13.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:39:13.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:13.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:15.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:16.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:16.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:16.003 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:39:16.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:39:16.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:16.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:17.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:18.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:18.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:18.082 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:39:18.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:39:18.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:18.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:18.503 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1382268391,v1:127.0.0.1:6803/1382268391] [v2:127.0.0.1:6804/1382268391,v1:127.0.0.1:6805/1382268391] exists,up 29c28616-2ee3-4b70-a401-f23f8969b42a 2024-09-10T17:39:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:39:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:39:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:39:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: get_asok_path osd.0 2024-09-10T17:39:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:39:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:39:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:39:18.505 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:18.505 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:18.505 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:39:18.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: CEPH_ARGS= 2024-09-10T17:39:18.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_max_backfills 2024-09-10T17:39:18.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-10T17:39:18.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-09-10T17:39:18.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: grep --quiet osd_max_backfills 2024-09-10T17:39:18.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:889: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-09-10T17:39:18.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:39:18.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:39:18.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:39:18.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:39:18.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:39:18.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:39:18.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-10T17:39:18.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2024-09-10T17:39:18.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:39:18.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:39:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:39:18.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:39:18.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-10T17:39:18.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:39:18.900 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:39:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T17:39:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-10T17:39:18.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-10T17:39:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T17:39:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:39:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:39:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:39:18.925 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:18.924+0000 7f4844530640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:18.927 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:18.926+0000 7f4844530640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:18.928 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:18.927+0000 7f4844530640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:19.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T17:39:19.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:39:19.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:39:19.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:39:19.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:39:19.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:19.333 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:39:19.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:39:19.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:19.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:19.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:20.279 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:20.278+0000 7f4844530640 -1 Falling back to public interface 2024-09-10T17:39:20.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:20.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:20.759 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:39:20.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:39:20.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:20.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:21.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:22.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:22.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:22.191 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:39:22.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:39:22.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:22.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:22.205 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:22.204+0000 7f4844530640 -1 osd.0 6 log_to_monitors true 2024-09-10T17:39:22.594 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/4059137406,v1:127.0.0.1:6803/4059137406] [v2:127.0.0.1:6804/4059137406,v1:127.0.0.1:6805/4059137406] exists,up 29c28616-2ee3-4b70-a401-f23f8969b42a 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: get_asok_path osd.0 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:22.596 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:22.597 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:39:22.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: CEPH_ARGS= 2024-09-10T17:39:22.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_max_backfills 2024-09-10T17:39:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-10T17:39:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:894: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: teardown td/ceph-helpers 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:39:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:39:22.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:39:22.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:39:22.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:39:22.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:39:22.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:39:22.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:39:22.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:39:22.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:39:22.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:39:22.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:39:22.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:39:22.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:39:22.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:39:22.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:39:22.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:39:22.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:22.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:22.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:39:22.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:39:22.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:39:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:39:22.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:39:22.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:39:22.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:39:22.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:39:22.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:39:22.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:39:22.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:39:22.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:39:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:39:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:39:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:39:22.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:39:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:39:22.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:39:22.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:39:22.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:39:22.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:22.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:22.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:39:22.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:39:22.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:39:22.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:39:22.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:39:22.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:22.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:22.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:39:22.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:904: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:39:22.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-10T17:39:23.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:39:23.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:39:23.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:39:23.260 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:39:23.260 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:23.260 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:23.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:39:23.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:39:23.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:39:23.454 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:23.455 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:23.455 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:39:23.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:39:23.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:39:23.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:39:23.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:39:23.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:39:23.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:39:23.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:39:23.533 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:39:23.533 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:39:23.533 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:39:23.534 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:39:23.534 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:23.534 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:23.534 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:39:23.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:39:23.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:39:23.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-09-10T17:39:23.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:39:23.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:39:23.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:39:23.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:39:23.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:39:23.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:39:24.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:39:24.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:39:24.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:39:24.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:39:24.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:24.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:24.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:39:24.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:39:24.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:39:24.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:907: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:39:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:24.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:39:24.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:39:24.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:39:24.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:39:24.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:39:24.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:39:24.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:39:24.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=09a8be91-d493-4f9f-b272-e8fab893bff8 2024-09-10T17:39:24.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 09a8be91-d493-4f9f-b272-e8fab893bff8' 2024-09-10T17:39:24.486 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 09a8be91-d493-4f9f-b272-e8fab893bff8 2024-09-10T17:39:24.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:39:24.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBMhOBmw77AHRAAedp711TIRNCIW+9KFK2loQ== 2024-09-10T17:39:24.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBMhOBmw77AHRAAedp711TIRNCIW+9KFK2loQ=="}' 2024-09-10T17:39:24.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 09a8be91-d493-4f9f-b272-e8fab893bff8 -i td/ceph-helpers/0/new.json 2024-09-10T17:39:24.760 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:39:24.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:39:24.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMhOBmw77AHRAAedp711TIRNCIW+9KFK2loQ== --osd-uuid 09a8be91-d493-4f9f-b272-e8fab893bff8 2024-09-10T17:39:24.799 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:24.798+0000 7f0d6efa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:24.829 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:24.828+0000 7f0d6efa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:24.830 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:24.829+0000 7f0d6efa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:24.830 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:24.829+0000 7f0d6efa4640 -1 bdev(0x55f397d17c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:39:24.830 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:24.829+0000 7f0d6efa4640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:39:30.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:39:30.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:39:30.275 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:39:30.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:39:30.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:39:30.861 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:39:30.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:39:30.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:39:30.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:39:30.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:39:30.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:39:30.889 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:30.888+0000 7f4af14b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:30.891 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:30.890+0000 7f4af14b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:30.892 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:30.891+0000 7f4af14b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:31.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:39:31.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:39:31.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:39:31.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:39:31.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:39:31.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:31.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:39:31.289 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:39:31.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:31.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:31.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:31.980 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:31.979+0000 7f4af14b0640 -1 Falling back to public interface 2024-09-10T17:39:32.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:32.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:32.715 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:39:32.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:39:32.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:32.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:33.668 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:33.667+0000 7f4af14b0640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:39:34.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:34.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:34.133 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:39:34.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:39:34.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:34.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:34.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:35.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:35.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:35.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:39:35.627 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:39:35.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:35.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:36.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:37.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:37.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:37.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:39:37.022 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:39:37.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:37.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:37.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:38.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:38.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:38.460 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:39:38.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:39:38.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:38.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:38.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:39.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:39.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:39.883 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:39:39.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:39:39.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:39.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:40.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:41.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:41.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:41.439 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:39:41.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:39:41.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:41.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:41.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:42.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:42.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:42.850 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:39:42.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:39:42.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:42.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:43.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:44.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:44.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:44.247 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:39:44.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:39:44.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:44.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:44.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:45.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:45.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:45.709 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:39:45.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:39:45.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:45.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:46.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:47.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:47.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:47.302 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:39:47.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:39:47.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:47.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:47.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:48.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:48.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:48.847 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:39:48.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:39:48.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:48.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:49.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:50.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:50.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:50.889 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:39:50.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:39:50.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:50.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:51.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:52.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:52.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:52.726 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:39:52.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:39:52.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:52.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:53.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:54.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:54.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:54.577 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:39:54.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:39:54.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:54.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:55.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:56.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:56.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:56.911 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:39:56.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:39:56.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:56.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:57.344 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2859223239,v1:127.0.0.1:6803/2859223239] [v2:127.0.0.1:6804/2859223239,v1:127.0.0.1:6805/2859223239] exists,up 09a8be91-d493-4f9f-b272-e8fab893bff8 2024-09-10T17:39:57.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:39:57.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:39:57.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:39:57.345 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-10T17:39:57.345 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:39:57.345 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:39:57.346 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:39:57.346 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:57.346 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:57.346 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:39:57.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-10T17:39:57.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_max_backfills 2024-09-10T17:39:57.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-09-10T17:39:57.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-09-10T17:39:57.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-09-10T17:39:57.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:912: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-09-10T17:39:57.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:39:57.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:39:57.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:39:57.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:39:57.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:39:57.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:39:57.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: ceph osd down 0 2024-09-10T17:39:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-10T17:39:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-09-10T17:39:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2024-09-10T17:39:58.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:39:58.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:39:58.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:39:58.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:58.147 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:39:58.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:39:58.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:58.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2024-09-10T17:39:58.588 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/2859223239,v1:127.0.0.1:6803/2859223239] [v2:127.0.0.1:6804/2859223239,v1:127.0.0.1:6805/2859223239] exists 09a8be91-d493-4f9f-b272-e8fab893bff8 2024-09-10T17:39:58.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:39:58.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:39:58.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:39:58.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:39:58.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:39:58.594 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:39:58.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T17:39:58.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-10T17:39:58.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-10T17:39:58.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T17:39:58.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:39:58.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:39:58.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:39:58.617 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:58.616+0000 7f01e3430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:58.636 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:58.635+0000 7f01e3430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:58.637 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:39:58.636+0000 7f01e3430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:59.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T17:39:59.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:39:59.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:39:59.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:39:59.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:39:59.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:59.042 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:39:59.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:39:59.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:59.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:39:59.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:00.002 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:00.001+0000 7f01e3430640 -1 Falling back to public interface 2024-09-10T17:40:00.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:00.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:00.468 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:40:00.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:40:00.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:00.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:00.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:01.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:01.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:01.906 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:40:01.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:40:01.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:01.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:01.958 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:01.957+0000 7f01e3430640 -1 osd.0 6 log_to_monitors true 2024-09-10T17:40:02.323 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/3775703057,v1:127.0.0.1:6803/3775703057] [v2:127.0.0.1:6804/3775703057,v1:127.0.0.1:6805/3775703057] exists,up 09a8be91-d493-4f9f-b272-e8fab893bff8 2024-09-10T17:40:02.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:02.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:02.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:02.324 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-10T17:40:02.324 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:40:02.324 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:40:02.324 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:02.325 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:02.325 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:02.325 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:40:02.325 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-10T17:40:02.325 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_max_backfills 2024-09-10T17:40:02.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-09-10T17:40:02.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-10T17:40:02.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-09-10T17:40:02.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:40:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:40:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:40:02.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:02.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:02.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:02.515 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:40:02.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:40:02.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:40:02.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:40:02.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:40:02.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:40:02.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:02.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:40:02.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:40:02.518 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:02.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:40:02.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:40:02.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:40:02.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:40:02.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:02.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:02.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:40:02.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:40:02.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:40:02.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:40:02.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-09-10T17:40:02.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-09-10T17:40:02.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-09-10T17:40:02.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:02.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:02.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:02.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:40:02.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:40:02.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:40:02.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:40:02.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:40:02.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:40:02.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:02.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:40:02.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:40:02.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:02.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:40:02.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:40:02.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:40:02.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:40:02.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:02.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:02.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:40:02.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:40:02.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:40:02.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:40:02.537 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:40:02.537 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:02.537 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:02.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:40:02.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:40:02.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:40:02.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:40:02.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:40:02.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-09-10T17:40:02.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:40:02.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:40:02.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:40:02.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:40:02.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:40:02.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-10T17:40:02.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:40:02.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:02.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:02.986 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:02.986 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:02.986 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:02.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:02.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:40:03.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:40:03.180 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:03.181 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:03.181 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:03.181 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:40:03.181 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:40:03.181 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:40:03.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:40:03.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:40:03.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:40:03.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:40:03.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:40:03.259 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:40:03.260 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:40:03.260 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:40:03.260 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:03.260 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:03.260 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:03.260 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:40:03.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:40:03.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:40:03.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-09-10T17:40:03.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:40:03.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:40:03.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:40:03.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:40:03.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:40:03.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:40:04.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:40:04.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:04.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:04.193 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:04.193 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:04.193 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:04.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:04.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:40:04.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:40:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-10T17:40:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:40:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:40:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:40:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:40:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:40:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:40:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:04.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-10T17:40:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:40:04.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:40:04.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=83e0e48a-1fe8-4ada-85b7-b7f9d3cc83a1 2024-09-10T17:40:04.226 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 83e0e48a-1fe8-4ada-85b7-b7f9d3cc83a1 2024-09-10T17:40:04.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 83e0e48a-1fe8-4ada-85b7-b7f9d3cc83a1' 2024-09-10T17:40:04.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:40:04.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB0hOBm36xqDhAAS1nE7eX4AQc7NBSeHMv74w== 2024-09-10T17:40:04.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB0hOBm36xqDhAAS1nE7eX4AQc7NBSeHMv74w=="}' 2024-09-10T17:40:04.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 83e0e48a-1fe8-4ada-85b7-b7f9d3cc83a1 -i td/ceph-helpers/0/new.json 2024-09-10T17:40:04.536 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:40:04.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:40:04.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQB0hOBm36xqDhAAS1nE7eX4AQc7NBSeHMv74w== --osd-uuid 83e0e48a-1fe8-4ada-85b7-b7f9d3cc83a1 2024-09-10T17:40:04.577 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:04.576+0000 7f37a28b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:04.596 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:04.595+0000 7f37a28b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:04.597 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:04.596+0000 7f37a28b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:04.598 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:04.597+0000 7f37a28b6640 -1 bdev(0x561a2ab69c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:40:04.598 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:04.597+0000 7f37a28b6640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:40:10.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:40:10.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:40:10.060 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:40:10.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:40:10.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:40:10.625 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:40:10.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:40:10.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-10T17:40:10.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:40:10.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:40:10.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:40:10.654 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:10.653+0000 7f5c7d00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:10.667 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:10.666+0000 7f5c7d00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:10.668 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:10.667+0000 7f5c7d00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:11.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:40:11.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:40:11.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:40:11.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:40:11.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:40:11.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:11.081 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:40:11.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:40:11.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:11.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:11.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:12.260 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:12.259+0000 7f5c7d00c640 -1 Falling back to public interface 2024-09-10T17:40:12.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:12.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:12.532 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:40:12.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:40:12.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:12.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:12.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:13.593 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:13.592+0000 7f5c7d00c640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:40:13.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:13.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:13.959 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:40:13.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:40:13.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:13.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:14.373 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1735058988,v1:127.0.0.1:6803/1735058988] [v2:127.0.0.1:6804/1735058988,v1:127.0.0.1:6805/1735058988] exists,up 83e0e48a-1fe8-4ada-85b7-b7f9d3cc83a1 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:40:14.375 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-10T17:40:14.376 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-10T17:40:14.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-10T17:40:14.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-10T17:40:14.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-10T17:40:14.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:940: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-09-10T17:40:14.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:14.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:14.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:14.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:14.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:14.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:14.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:941: test_activate_osd_skip_benchmark: ceph osd down 0 2024-09-10T17:40:15.266 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-10T17:40:15.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-09-10T17:40:15.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2024-09-10T17:40:15.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:40:15.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:40:15.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:40:15.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:15.278 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:40:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:40:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:15.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2024-09-10T17:40:15.702 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1735058988,v1:127.0.0.1:6803/1735058988] [v2:127.0.0.1:6804/1735058988,v1:127.0.0.1:6805/1735058988] exists 83e0e48a-1fe8-4ada-85b7-b7f9d3cc83a1 2024-09-10T17:40:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:946: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-10T17:40:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2024-09-10T17:40:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:40:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T17:40:15.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:15.707 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:40:15.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-10T17:40:15.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:40:15.709 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:40:15.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T17:40:15.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-10T17:40:15.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-10T17:40:15.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T17:40:15.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:40:15.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:40:15.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:40:15.734 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:15.732+0000 7f471b09c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:15.735 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:15.734+0000 7f471b09c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:15.736 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:15.735+0000 7f471b09c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:16.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T17:40:16.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:40:16.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:40:16.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:40:16.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:40:16.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:16.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:40:16.095 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:40:16.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:16.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:16.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:17.074 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:17.073+0000 7f471b09c640 -1 Falling back to public interface 2024-09-10T17:40:17.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:17.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:17.524 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:40:17.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:40:17.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:17.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:17.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:18.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:18.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:18.913 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:40:18.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:40:18.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:18.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:19.155 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:19.154+0000 7f471b09c640 -1 osd.0 5 log_to_monitors true 2024-09-10T17:40:19.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:20.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:20.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:20.328 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:40:20.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:40:20.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:20.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:20.717 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/210808363,v1:127.0.0.1:6803/210808363] [v2:127.0.0.1:6804/210808363,v1:127.0.0.1:6805/210808363] exists,up 83e0e48a-1fe8-4ada-85b7-b7f9d3cc83a1 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:20.719 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:20.720 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:40:20.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-10T17:40:20.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-10T17:40:20.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-10T17:40:20.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-10T17:40:20.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-10T17:40:20.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:953: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:954: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:20.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:20.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:40:20.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:40:20.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:40:20.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:40:20.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:40:20.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:40:20.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:20.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:40:20.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:40:20.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:20.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:40:20.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:40:20.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:40:21.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:40:21.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:21.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:21.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:40:21.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:40:21.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:40:21.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:40:21.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_destroy_osd td/ceph-helpers 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: test_destroy_osd: local dir=td/ceph-helpers 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: test_destroy_osd: setup td/ceph-helpers 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:21.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:21.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:21.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:40:21.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:40:21.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:40:21.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:40:21.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:40:21.007 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:40:21.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:21.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:40:21.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:40:21.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:21.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:40:21.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:40:21.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:40:21.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:40:21.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:21.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:21.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:40:21.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:40:21.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:40:21.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:40:21.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:40:21.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:21.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:21.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:40:21.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: test_destroy_osd: run_mon td/ceph-helpers a 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:40:21.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-10T17:40:21.476 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:40:21.476 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:21.476 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:21.476 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:21.476 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:21.476 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:21.476 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:21.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:40:21.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:40:21.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:40:21.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:40:21.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:40:21.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:40:21.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:40:21.669 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:40:21.670 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:40:21.670 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:40:21.670 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:21.670 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:21.670 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:21.670 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:40:21.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:40:21.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:40:21.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:40:21.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:40:21.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: test_destroy_osd: run_mgr td/ceph-helpers x 2024-09-10T17:40:21.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:40:21.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:40:21.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:40:21.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:40:21.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:40:21.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:40:22.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:40:22.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:22.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:22.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:22.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:40:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:40:22.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: test_destroy_osd: run_osd td/ceph-helpers 0 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:40:22.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:40:22.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:40:22.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:40:22.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d4b0ffd6-c548-40b2-a7d3-e844baa9dea1 2024-09-10T17:40:22.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d4b0ffd6-c548-40b2-a7d3-e844baa9dea1' 2024-09-10T17:40:22.715 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 d4b0ffd6-c548-40b2-a7d3-e844baa9dea1 2024-09-10T17:40:22.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:40:22.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGhOBmkbSNKxAAqpGB9wN1Ly47b6mkzkieQQ== 2024-09-10T17:40:22.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGhOBmkbSNKxAAqpGB9wN1Ly47b6mkzkieQQ=="}' 2024-09-10T17:40:22.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d4b0ffd6-c548-40b2-a7d3-e844baa9dea1 -i td/ceph-helpers/0/new.json 2024-09-10T17:40:23.000 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:40:23.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:40:23.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGhOBmkbSNKxAAqpGB9wN1Ly47b6mkzkieQQ== --osd-uuid d4b0ffd6-c548-40b2-a7d3-e844baa9dea1 2024-09-10T17:40:23.041 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:23.040+0000 7f603e228640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:23.069 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:23.068+0000 7f603e228640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:23.070 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:23.069+0000 7f603e228640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:23.071 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:23.069+0000 7f603e228640 -1 bdev(0x562d7f83bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:40:23.071 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:23.069+0000 7f603e228640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:40:28.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:40:28.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:40:28.804 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:40:28.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:40:28.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:40:29.347 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:40:29.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:40:29.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:40:29.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:40:29.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:40:29.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:40:29.376 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:29.376+0000 7f9146011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:29.378 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:29.378+0000 7f9146011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:29.379 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:29.379+0000 7f9146011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:29.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:40:29.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:40:29.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:40:29.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:40:29.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:40:29.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:29.776 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:40:29.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:40:29.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:29.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:30.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:30.253 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:30.236+0000 7f9146011640 -1 Falling back to public interface 2024-09-10T17:40:31.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:31.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:31.194 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:40:31.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:40:31.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:31.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:31.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:32.254 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:40:32.253+0000 7f9146011640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:40:32.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:32.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:32.628 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:40:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:40:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:33.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:34.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:34.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:34.192 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:40:34.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:40:34.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:34.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:34.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:35.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:35.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:35.622 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:40:35.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:40:35.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:35.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:36.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:37.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:37.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:37.135 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:40:37.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:40:37.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:37.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:37.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:38.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:38.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:38.574 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:40:38.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:40:38.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:38.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:39.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:40.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:40.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:40.005 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:40:40.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:40:40.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:40.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:40.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:41.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:41.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:41.434 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:40:41.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:40:41.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:41.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:41.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:42.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:42.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:42.835 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:40:42.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:40:42.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:42.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:43.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:44.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:44.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:44.307 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:40:44.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:40:44.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:44.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:45.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:46.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:46.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:46.320 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:40:46.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:40:46.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:46.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:47.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:48.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:48.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:48.010 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:40:48.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:40:48.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:48.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:48.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:49.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:49.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:49.870 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:40:49.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:40:49.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:49.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:50.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:51.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:51.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:51.567 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:40:51.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:40:51.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:51.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:52.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:53.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:53.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:53.350 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:40:53.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:40:53.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:53.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:54.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:55.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:55.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:55.343 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:40:55.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:40:55.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:55.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:40:55.749 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1417279287,v1:127.0.0.1:6803/1417279287] [v2:127.0.0.1:6804/1417279287,v1:127.0.0.1:6805/1417279287] exists,up d4b0ffd6-c548-40b2-a7d3-e844baa9dea1 2024-09-10T17:40:55.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:55.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:55.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:55.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:793: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-09-10T17:40:55.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:775: destroy_osd: local dir=td/ceph-helpers 2024-09-10T17:40:55.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:776: destroy_osd: local id=0 2024-09-10T17:40:55.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:778: destroy_osd: ceph osd out osd.0 2024-09-10T17:40:56.246 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already out. 2024-09-10T17:40:56.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:779: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-10T17:40:56.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:56.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:56.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:56.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:56.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:56.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:56.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:780: destroy_osd: ceph osd down osd.0 2024-09-10T17:40:57.025 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-10T17:40:57.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:781: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-09-10T17:40:57.451 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 does not exist 2024-09-10T17:40:57.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:782: destroy_osd: teardown td/ceph-helpers/0 2024-09-10T17:40:57.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers/0 2024-09-10T17:40:57.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:40:57.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-09-10T17:40:57.464 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:57.464 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:57.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:57.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:57.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:57.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:57.465 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:40:57.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:40:57.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:40:57.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:40:57.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:40:57.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:40:57.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:57.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:40:57.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:40:57.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:57.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:40:57.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:40:57.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers/0 2024-09-10T17:40:57.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:40:57.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:57.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:57.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:40:57.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:40:57.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:40:57.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:783: destroy_osd: rm -fr td/ceph-helpers/0 2024-09-10T17:40:57.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: ceph osd dump 2024-09-10T17:40:57.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: grep 'osd. ' 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:795: test_destroy_osd: teardown td/ceph-helpers 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:57.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:57.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:57.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:40:57.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:40:57.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:40:57.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:40:57.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:40:57.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:40:57.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:57.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:40:57.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:40:57.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:57.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:40:57.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:40:57.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:40:57.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:40:57.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:57.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:57.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:40:57.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:40:57.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:40:57.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:40:57.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_display_logs td/ceph-helpers 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: test_display_logs: local dir=td/ceph-helpers 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: test_display_logs: setup td/ceph-helpers 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:57.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:57.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:57.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:40:57.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:40:57.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:40:57.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:40:57.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:40:57.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:40:57.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:57.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:40:57.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:40:57.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:40:57.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:40:57.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:40:57.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:40:57.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:40:57.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:57.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:57.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:40:57.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:40:57.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:40:57.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:40:57.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:40:57.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:57.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:57.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:40:57.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:40:57.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:40:57.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:40:57.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:40:57.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: test_display_logs: run_mon td/ceph-helpers a 2024-09-10T17:40:57.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:40:57.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:40:57.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:40:57.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:40:57.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:40:57.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-10T17:40:58.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:40:58.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:58.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:58.467 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:58.467 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:58.467 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:58.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:40:58.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:40:58.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:58.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:40:58.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:40:58.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:58.741 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:40:58.742 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:40:58.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:40:58.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:40:58.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: test_display_logs: kill_daemons td/ceph-helpers 2024-09-10T17:40:58.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:58.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:58.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:58.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:58.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:00.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:00.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: test_display_logs: display_logs td/ceph-helpers 2024-09-10T17:41:00.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: display_logs: local dir=td/ceph-helpers 2024-09-10T17:41:00.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-09-10T17:41:00.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: display_logs: read file 2024-09-10T17:41:00.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-09-10T17:41:00.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: display_logs: cat td/ceph-helpers/mon.a.log 2024-09-10T17:41:00.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: display_logs: read file 2024-09-10T17:41:00.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-09-10T17:41:00.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: test_display_logs: teardown td/ceph-helpers 2024-09-10T17:41:00.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:41:00.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:00.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:41:00.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:00.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:00.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:00.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:00.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:00.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:00.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:00.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:00.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:00.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:00.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:00.136 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:00.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:00.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:00.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:00.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:00.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:00.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:00.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:41:00.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:00.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:00.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:00.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:41:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:41:00.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:00.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:41:00.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:00.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:00.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:00.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:00.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:00.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:00.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:00.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:00.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:00.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:00.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:00.148 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:00.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:00.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:00.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:00.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:00.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:41:00.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:00.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:00.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:41:00.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:41:00.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:00.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:00.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:41:00.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:41:00.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:41:00.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:41:00.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:41:00.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-09-10T17:41:00.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:41:00.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:41:00.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:41:00.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:41:00.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:41:00.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-10T17:41:00.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:41:00.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:00.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:00.581 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:00.581 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:00.581 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:00.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:00.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:41:00.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:41:00.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:41:00.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:00.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:00.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:41:00.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:41:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:00.786 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:41:00.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:00.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:41:00.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:41:00.938 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:00.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:41:00.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-09-10T17:41:00.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:41:00.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:41:00.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:41:00.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:41:00.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:41:00.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:41:01.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:41:01.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:01.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:01.813 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:01.813 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:01.813 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:01.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:01.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:41:01.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:41:01.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-09-10T17:41:01.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: erasure_code_plugin_exists: local plugin=jerasure 2024-09-10T17:41:01.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: local status 2024-09-10T17:41:01.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local grepstr 2024-09-10T17:41:01.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: local s 2024-09-10T17:41:01.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: case `uname` in 2024-09-10T17:41:01.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: uname 2024-09-10T17:41:01.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-09-10T17:41:01.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-09-10T17:41:02.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: s= 2024-09-10T17:41:02.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status=0 2024-09-10T17:41:02.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-10T17:41:02.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-10T17:41:02.474 INFO:tasks.workunit.client.0.smithi155.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-10T17:41:02.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: return 0 2024-09-10T17:41:02.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-09-10T17:41:02.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: erasure_code_plugin_exists: local plugin=FAKE 2024-09-10T17:41:02.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: local status 2024-09-10T17:41:02.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local grepstr 2024-09-10T17:41:02.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: local s 2024-09-10T17:41:02.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: case `uname` in 2024-09-10T17:41:02.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: uname 2024-09-10T17:41:02.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-09-10T17:41:02.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-09-10T17:41:02.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-09-10T17:41:02.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status=5 2024-09-10T17:41:02.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-09-10T17:41:02.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-09-10T17:41:02.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-09-10T17:41:02.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: return 5 2024-09-10T17:41:02.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-09-10T17:41:02.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:41:02.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:02.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:41:02.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:02.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:02.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:02.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:02.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:02.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:02.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:02.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:02.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:02.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:02.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:02.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:02.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:02.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:02.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:02.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:02.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:41:02.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:02.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:02.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:02.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:41:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:41:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_expect_failure td/ceph-helpers 2024-09-10T17:41:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_expect_failure: local dir=td/ceph-helpers 2024-09-10T17:41:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: test_expect_failure: setup td/ceph-helpers 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:02.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:02.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:02.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:02.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:02.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:02.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:02.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:02.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:02.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:02.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:02.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:02.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:02.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:02.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:41:02.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:02.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:02.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:02.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:41:02.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:02.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:02.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:41:02.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:41:02.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:02.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:02.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:41:02.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:41:02.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/ceph-helpers 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=FAIL 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T17:41:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-09-10T17:41:02.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T17:41:02.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T17:41:02.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-10T17:41:02.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-10T17:41:02.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-09-10T17:41:02.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/ceph-helpers 2024-09-10T17:41:02.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T17:41:02.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=FAIL 2024-09-10T17:41:02.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T17:41:02.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T17:41:02.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-09-10T17:41:02.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: expect_failure: success=true 2024-09-10T17:41:02.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: true 2024-09-10T17:41:02.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: cat td/ceph-helpers/out 2024-09-10T17:41:02.776 INFO:tasks.workunit.client.0.smithi155.stderr:FAIL 2024-09-10T17:41:02.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: return 1 2024-09-10T17:41:02.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1997: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-10T17:41:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1999: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-09-10T17:41:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/ceph-helpers 2024-09-10T17:41:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T17:41:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=FAIL 2024-09-10T17:41:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T17:41:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T17:41:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-09-10T17:41:02.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T17:41:02.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T17:41:02.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-10T17:41:02.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: cat td/ceph-helpers/out 2024-09-10T17:41:02.779 INFO:tasks.workunit.client.0.smithi155.stderr:UNEXPECTED 2024-09-10T17:41:02.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: return 1 2024-09-10T17:41:02.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2000: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-10T17:41:02.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: test_expect_failure: teardown td/ceph-helpers 2024-09-10T17:41:02.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:41:02.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:02.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:41:02.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:02.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:02.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:02.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:02.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:02.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:02.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:02.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:02.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:02.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:02.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:02.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:02.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:02.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:02.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:02.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:02.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:41:02.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:02.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:02.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:02.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_flush_pg_stats td/ceph-helpers 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: test_flush_pg_stats: local dir=td/ceph-helpers 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_flush_pg_stats: setup td/ceph-helpers 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:41:02.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:02.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:02.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:02.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:02.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:02.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:02.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:02.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:02.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:02.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:02.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:02.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:02.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:02.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:02.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:02.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:41:02.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:02.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:02.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:41:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:41:02.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:41:02.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:02.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:02.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:41:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:41:03.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:41:03.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:03.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:03.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:03.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:03.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:03.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:03.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:41:03.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:03.469 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:41:03.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:03.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:03.547 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:03.548 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:03.548 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:03.548 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:41:03.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:03.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:41:03.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-09-10T17:41:03.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:41:03.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:41:03.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:41:03.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:41:03.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:41:03.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:41:04.470 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:41:04.470 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:04.470 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:04.471 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:04.471 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:04.471 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:04.471 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:04.471 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:41:04.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:41:04.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:41:04.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:41:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ac74f67d-e2af-4a0d-ab98-63e73be6cb5b 2024-09-10T17:41:04.505 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 ac74f67d-e2af-4a0d-ab98-63e73be6cb5b 2024-09-10T17:41:04.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ac74f67d-e2af-4a0d-ab98-63e73be6cb5b' 2024-09-10T17:41:04.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:41:04.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCwhOBmPGYJHxAAh6Cg68Lszylym+YXum46eA== 2024-09-10T17:41:04.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCwhOBmPGYJHxAAh6Cg68Lszylym+YXum46eA=="}' 2024-09-10T17:41:04.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ac74f67d-e2af-4a0d-ab98-63e73be6cb5b -i td/ceph-helpers/0/new.json 2024-09-10T17:41:04.790 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:41:04.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:41:04.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwhOBmPGYJHxAAh6Cg68Lszylym+YXum46eA== --osd-uuid ac74f67d-e2af-4a0d-ab98-63e73be6cb5b 2024-09-10T17:41:04.831 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:04.829+0000 7f7f270ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:04.859 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:04.858+0000 7f7f270ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:04.860 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:04.859+0000 7f7f270ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:04.861 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:04.859+0000 7f7f270ef640 -1 bdev(0x55fd57df5c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:41:04.861 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:04.860+0000 7f7f270ef640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:41:10.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:41:10.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:41:10.383 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:41:10.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:41:10.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:41:10.949 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:41:10.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:41:10.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:41:10.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:41:10.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:41:10.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:41:10.977 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:10.975+0000 7fc41fa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:10.978 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:10.977+0000 7fc41fa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:10.979 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:10.978+0000 7fc41fa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:41:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:41:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:41:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:41:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:41:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:41:11.368 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:41:11.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:11.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:11.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:12.084 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:12.083+0000 7fc41fa2c640 -1 Falling back to public interface 2024-09-10T17:41:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:12.836 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:41:12.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:41:12.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:12.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:13.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:13.947 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:13.946+0000 7fc41fa2c640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:41:14.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:14.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:14.271 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:41:14.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:41:14.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:14.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:14.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:15.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:15.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:15.673 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:41:15.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:41:15.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:15.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:16.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:17.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:17.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:17.098 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:41:17.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:41:17.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:17.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:17.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:18.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:18.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:18.500 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:41:18.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:41:18.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:18.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:18.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:19.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:19.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:19.951 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:41:19.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:41:19.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:19.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:20.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:21.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:21.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:21.390 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:41:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:41:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:21.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:22.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:22.805 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:41:22.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:41:22.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:22.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:23.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:24.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:24.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:24.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:41:24.251 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:41:24.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:24.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:24.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:25.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:25.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:25.673 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:41:25.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:41:25.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:25.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:26.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:27.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:27.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:27.608 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:41:27.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:41:27.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:27.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:28.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:29.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:29.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:29.600 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:41:29.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:41:29.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:29.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:30.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:31.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:31.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:31.002 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:41:31.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:41:31.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:31.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:31.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:32.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:32.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:32.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:41:32.791 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:41:32.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:32.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:33.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:34.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:34.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:34.791 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:41:34.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:41:34.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:34.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:35.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:36.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:36.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:36.734 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:41:36.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:41:36.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:36.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:37.130 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1248319734,v1:127.0.0.1:6803/1248319734] [v2:127.0.0.1:6804/1248319734,v1:127.0.0.1:6805/1248319734] exists,up ac74f67d-e2af-4a0d-ab98-63e73be6cb5b 2024-09-10T17:41:37.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:41:37.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:41:37.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:41:37.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_flush_pg_stats: create_rbd_pool 2024-09-10T17:41:37.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:41:37.544 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:41:37.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:41:37.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:41:38.080 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:41:38.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:41:39.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:41:40.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-09-10T17:41:40.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: test_flush_pg_stats: flush_pg_stats 2024-09-10T17:41:40.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:41:40.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:41:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:41:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:41:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:40.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:41:40.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836485 2024-09-10T17:41:40.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836485 2024-09-10T17:41:40.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836485' 2024-09-10T17:41:40.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:40.859 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836485 2024-09-10T17:41:40.859 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:40.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:41:40.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836485 2024-09-10T17:41:40.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:40.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836485 2024-09-10T17:41:40.861 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836485 2024-09-10T17:41:40.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-10T17:41:40.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:41:41.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-10T17:41:41.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:41:42.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:41:42.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:41:42.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-10T17:41:42.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:41:43.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T17:41:43.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:41:44.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-10T17:41:44.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-09-10T17:41:44.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_flush_pg_stats: ceph df detail --format=json 2024-09-10T17:41:44.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-09-10T17:41:44.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_flush_pg_stats: stored=6755 2024-09-10T17:41:44.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: ceph df detail --format=json 2024-09-10T17:41:44.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: stored_raw=6755 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: test_flush_pg_stats: test 6755 -gt 0 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: test_flush_pg_stats: test 6755 == 6755 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: test_flush_pg_stats: teardown td/ceph-helpers 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:45.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:45.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:45.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:45.363 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:45.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:45.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:45.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:45.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:45.365 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:45.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:45.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:45.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:45.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:45.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:45.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:45.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:41:45.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:45.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:45.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:45.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_config td/ceph-helpers 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: test_get_config: local dir=td/ceph-helpers 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: test_get_config: setup td/ceph-helpers 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:45.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:41:45.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:45.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:45.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:45.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:45.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:45.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:45.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:45.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:45.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:45.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:45.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:45.380 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:45.381 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:45.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:45.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:45.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:41:45.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:45.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:45.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:45.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:41:45.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:45.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:45.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:41:45.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:41:45.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:45.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:45.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:41:45.388 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:41:45.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:41:45.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:41:45.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:41:45.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1131: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:41:45.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:41:45.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:41:45.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:41:45.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:41:45.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:41:45.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:41:45.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:41:45.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:45.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:45.903 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:45.903 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:45.903 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:45.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:45.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:41:46.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:41:46.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:41:46.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:46.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:46.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:41:46.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:41:46.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:46.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:46.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:46.098 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:46.098 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:46.098 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:46.098 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:41:46.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:46.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:41:46.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:41:46.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:46.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:41:46.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: get_config mon a osd_pool_default_size 2024-09-10T17:41:46.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:46.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_pool_default_size 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_pool_default_size 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:46.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get osd_pool_default_size 2024-09-10T17:41:46.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: test 1 = 1 2024-09-10T17:41:46.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: test_get_config: run_mgr td/ceph-helpers x 2024-09-10T17:41:46.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:41:46.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:41:46.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:41:46.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:41:46.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:41:46.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:41:47.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:41:47.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:47.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:47.184 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:47.184 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:47.184 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:47.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:47.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:41:47.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:41:47.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-09-10T17:41:47.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:41:47.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:41:47.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:41:47.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:41:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:41:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:41:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:41:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:41:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_max_scrubs=3 2024-09-10T17:41:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:41:47.218 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:41:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e2789fe-3d38-4949-b8d8-69e81c522db0 2024-09-10T17:41:47.218 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 9e2789fe-3d38-4949-b8d8-69e81c522db0 2024-09-10T17:41:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9e2789fe-3d38-4949-b8d8-69e81c522db0' 2024-09-10T17:41:47.219 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:41:47.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbhOBmhuLoDRAA2Jpsg5KbOqbaBl0hP2dGcg== 2024-09-10T17:41:47.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbhOBmhuLoDRAA2Jpsg5KbOqbaBl0hP2dGcg=="}' 2024-09-10T17:41:47.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e2789fe-3d38-4949-b8d8-69e81c522db0 -i td/ceph-helpers/0/new.json 2024-09-10T17:41:47.511 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:41:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:41:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQDbhOBmhuLoDRAA2Jpsg5KbOqbaBl0hP2dGcg== --osd-uuid 9e2789fe-3d38-4949-b8d8-69e81c522db0 2024-09-10T17:41:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:47.550+0000 7f2b59418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:47.579+0000 7f2b59418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:47.580+0000 7f2b59418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:47.580+0000 7f2b59418640 -1 bdev(0x562f105dfc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:41:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:47.580+0000 7f2b59418640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:41:53.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:41:53.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:41:53.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:41:53.329 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:41:53.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:41:53.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:41:53.875 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:41:53.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-09-10T17:41:53.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:41:53.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:41:53.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:41:53.902 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:53.901+0000 7fbd0e41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:53.907 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:53.906+0000 7fbd0e41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:53.908 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:53.907+0000 7fbd0e41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:54.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:41:54.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:41:54.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:41:54.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:41:54.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:41:54.297 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:41:54.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:54.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:41:54.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:54.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:54.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:55.517 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:55.516+0000 7fbd0e41c640 -1 Falling back to public interface 2024-09-10T17:41:55.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:55.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:55.723 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:41:55.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:41:55.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:55.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:56.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:57.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:57.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:57.150 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:41:57.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:41:57.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:57.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:57.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:57.750 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:41:57.749+0000 7fbd0e41c640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:41:58.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:58.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:58.545 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:41:58.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:41:58.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:58.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:58.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:59.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:59.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:59.993 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:41:59.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:41:59.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:59.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:00.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:01.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:01.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:01.499 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:42:01.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:42:01.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:01.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:01.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:02.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:02.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:02.896 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:42:02.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:42:02.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:02.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:03.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:04.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:04.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:04.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:42:04.292 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:42:04.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:04.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:04.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:05.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:05.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:05.702 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:42:05.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:42:05.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:05.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:06.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:07.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:07.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:07.103 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:42:07.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:42:07.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:07.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:07.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:08.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:08.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:08.514 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:42:08.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:42:08.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:08.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:09.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:10.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:10.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:10.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:42:10.127 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:42:10.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:10.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:10.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:11.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:11.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:11.706 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:42:11.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:42:11.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:11.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:12.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:13.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:13.507 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:42:13.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:13.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:42:13.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:13.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:14.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:15.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:15.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:15.227 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:42:15.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:42:15.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:15.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:15.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:16.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:16.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:16.644 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:42:16.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:42:16.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:16.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:17.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:18.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:18.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:18.379 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:42:18.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:42:18.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:18.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:19.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:20.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:20.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:20.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:42:20.442 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:42:20.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:20.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:20.884 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2628279813,v1:127.0.0.1:6803/2628279813] [v2:127.0.0.1:6804/2628279813,v1:127.0.0.1:6805/2628279813] exists,up 9e2789fe-3d38-4949-b8d8-69e81c522db0 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: get_config osd 0 osd_max_scrubs 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_max_scrubs 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_max_scrubs 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:42:20.886 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:42:20.887 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:42:20.887 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:20.887 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:20.887 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:42:20.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:42:20.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_max_scrubs 2024-09-10T17:42:20.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: test 3 = 3 2024-09-10T17:42:20.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: teardown td/ceph-helpers 2024-09-10T17:42:20.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:42:20.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:42:20.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:42:20.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:42:20.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:42:20.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:42:20.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:42:20.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:42:21.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:42:21.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:42:21.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:42:21.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:42:21.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:42:21.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:42:21.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:42:21.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:42:21.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:42:21.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:42:21.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:42:21.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:42:21.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:42:21.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:42:21.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:42:21.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:21.088 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:21.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:42:21.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:42:21.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:42:21.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:42:21.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:42:21.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:42:21.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:42:21.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:42:21.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:42:21.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:42:21.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:42:21.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:42:21.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:42:21.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:42:21.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:42:21.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:42:21.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:42:21.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:42:21.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:42:21.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:42:21.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:42:21.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:42:21.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:21.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:21.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:42:21.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:42:21.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:42:21.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:42:21.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:42:21.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:21.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:21.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:42:21.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:42:21.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:42:21.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-10T17:42:21.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:42:21.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:42:21.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:42:21.581 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:42:21.581 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:21.581 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:21.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:42:21.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:42:21.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:42:21.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:42:21.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:42:21.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:42:21.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:42:21.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:42:21.766 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:42:21.767 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:42:21.767 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:42:21.767 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:42:21.767 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:21.767 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:21.767 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:42:21.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:42:21.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:42:21.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:21.850 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:42:21.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:42:21.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:42:21.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-09-10T17:42:21.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:42:21.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:42:21.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:42:21.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:42:21.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:42:21.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:42:22.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:42:22.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:42:22.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:42:22.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:42:22.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:22.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:22.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:42:22.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:42:22.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:42:22.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-09-10T17:42:22.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T17:42:22.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T17:42:22.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T17:42:22.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T17:42:22.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T17:42:22.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T17:42:23.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T17:42:23.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T17:42:23.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1341: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-09-10T17:42:23.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:42:23.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:42:23.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:42:23.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:42:23.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:42:23.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:42:23.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:42:23.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:42:23.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:42:23.148 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:42:23.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:42:23.149 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:42:23.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:42:23.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:42:23.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:42:23.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:42:23.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:42:23.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:42:23.152 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:42:23.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:42:23.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:42:23.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:42:23.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:42:23.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:23.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:23.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1524: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:42:23.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:42:23.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:42:23.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:42:23.159 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:42:23.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:42:23.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:42:23.160 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:42:23.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:42:23.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:42:23.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:42:23.162 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:42:23.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:42:23.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:42:23.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:42:23.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:42:23.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:23.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:42:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:42:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:42:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:42:23.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:42:23.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:23.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:23.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:42:23.169 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:42:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:42:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:42:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:42:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:42:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:42:23.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:42:23.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:42:23.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:42:23.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:42:23.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:42:23.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:42:23.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:42:23.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:42:23.638 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:42:23.638 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:23.638 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:23.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:42:23.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:42:23.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:42:23.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:42:23.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:42:23.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:42:23.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:42:23.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:42:23.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:42:23.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:42:23.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:42:23.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-09-10T17:42:23.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:42:23.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:42:23.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:42:23.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:42:23.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:42:23.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:42:24.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:42:24.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:42:24.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:42:24.827 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:42:24.827 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:24.827 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:24.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:42:24.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:42:24.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:42:24.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-09-10T17:42:24.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:42:24.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:42:24.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:42:24.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:42:24.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:42:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:42:24.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:42:24.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:42:24.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:42:24.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:42:24.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7517658a-011d-4464-8170-379b067519fc 2024-09-10T17:42:24.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7517658a-011d-4464-8170-379b067519fc' 2024-09-10T17:42:24.863 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 7517658a-011d-4464-8170-379b067519fc 2024-09-10T17:42:24.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:42:24.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAAheBmFaNVNBAA7w+VenVaSIgEqWIdEXUW8Q== 2024-09-10T17:42:24.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAAheBmFaNVNBAA7w+VenVaSIgEqWIdEXUW8Q=="}' 2024-09-10T17:42:24.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7517658a-011d-4464-8170-379b067519fc -i td/ceph-helpers/0/new.json 2024-09-10T17:42:25.155 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:42:25.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:42:25.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAheBmFaNVNBAA7w+VenVaSIgEqWIdEXUW8Q== --osd-uuid 7517658a-011d-4464-8170-379b067519fc 2024-09-10T17:42:25.195 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:25.194+0000 7f003c54d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:25.215 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:25.214+0000 7f003c54d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:25.216 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:25.215+0000 7f003c54d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:25.217 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:25.216+0000 7f003c54d640 -1 bdev(0x55e52ce85c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:42:25.217 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:25.216+0000 7f003c54d640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:42:30.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:42:30.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:42:30.989 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:42:30.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:42:30.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:42:31.536 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:42:31.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:42:31.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:42:31.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:42:31.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:42:31.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:42:31.564 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:31.563+0000 7fa94f4eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:31.568 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:31.566+0000 7fa94f4eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:31.569 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:31.568+0000 7fa94f4eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:31.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:42:31.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:42:31.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:42:31.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:42:31.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:42:31.982 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:42:31.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:31.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:42:31.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:31.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:32.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:32.933 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:32.932+0000 7fa94f4eb640 -1 Falling back to public interface 2024-09-10T17:42:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:33.409 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:42:33.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:42:33.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:33.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:33.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:34.553 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:42:34.552+0000 7fa94f4eb640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:42:34.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:34.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:34.843 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:42:34.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:42:34.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:34.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:35.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:36.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:36.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:36.334 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:42:36.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:42:36.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:36.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:36.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:37.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:37.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:37.730 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:42:37.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:42:37.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:37.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:38.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:39.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:39.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:39.153 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:42:39.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:42:39.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:39.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:39.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:40.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:40.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:40.697 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:42:40.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:42:40.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:40.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:41.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:42.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:42.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:42:42.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:42:42.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:42.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:42.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:43.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:43.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:43.523 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:42:43.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:42:43.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:43.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:43.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:44.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:44.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:44.944 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:42:44.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:42:44.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:44.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:45.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:46.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:46.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:46.470 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:42:46.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:42:46.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:46.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:48.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:48.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:48.354 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:42:48.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:42:48.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:48.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:48.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:49.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:49.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:49.910 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:42:49.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:42:49.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:49.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:50.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:51.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:51.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:51.594 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:42:51.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:42:51.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:51.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:52.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:53.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:53.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:53.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:42:53.734 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:42:53.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:53.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:54.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:55.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:55.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:55.516 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:42:55.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:42:55.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:55.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:56.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:57.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:57.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:57.451 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:42:57.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:42:57.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:57.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:57.874 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1047130004,v1:127.0.0.1:6803/1047130004] [v2:127.0.0.1:6804/1047130004,v1:127.0.0.1:6805/1047130004] exists,up 7517658a-011d-4464-8170-379b067519fc 2024-09-10T17:42:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:42:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:42:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:42:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: test_get_last_scrub_stamp: create_rbd_pool 2024-09-10T17:42:57.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:42:58.289 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:42:58.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:42:58.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:42:58.782 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:42:58.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:42:59.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:43:00.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: test_get_last_scrub_stamp: wait_for_clean 2024-09-10T17:43:00.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:43:00.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:43:00.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:43:00.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:43:00.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:43:00.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:43:00.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:43:00.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:43:00.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:43:00.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:43:00.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:43:00.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:43:00.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:43:00.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:43:00.893 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:43:01.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:43:01.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:43:01.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:43:01.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:43:01.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-10T17:43:01.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-10T17:43:01.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-10T17:43:01.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:43:01.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-10T17:43:01.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:43:01.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:43:01.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-10T17:43:01.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:43:01.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-10T17:43:01.447 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836487 2024-09-10T17:43:01.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-10T17:43:01.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:43:01.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-10T17:43:01.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:43:02.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:43:02.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:43:03.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-10T17:43:03.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:43:03.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:43:03.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:43:03.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:43:03.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:43:03.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:43:03.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:43:03.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:43:03.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:43:03.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:43:03.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:43:04.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:43:04.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:43:04.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:43:04.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:43:04.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:43:04.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:43:04.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:43:04.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-09-10T17:43:04.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:43:04.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:43:04.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:43:04.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:43:05.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: stamp=2024-09-10T17:42:58.623733+0000 2024-09-10T17:43:05.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: test -n 2024-09-10T17:42:58.623733+0000 2024-09-10T17:43:05.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1534: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-09-10T17:43:05.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:43:05.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:43:05.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:43:05.173 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:43:05.173 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:43:05.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:43:05.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:43:05.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:43:05.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:43:05.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:43:05.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:43:05.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:43:05.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:43:05.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:43:05.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:43:05.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:43:05.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:43:05.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:43:05.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:43:05.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:43:05.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:43:05.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:43:05.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:43:05.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:05.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:05.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:43:05.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:43:05.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:43:05.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_not_primary td/ceph-helpers 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: test_get_not_primary: local dir=td/ceph-helpers 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1231: test_get_not_primary: setup td/ceph-helpers 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:43:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:43:05.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:43:05.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:43:05.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:43:05.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:43:05.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:43:05.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:43:05.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:43:05.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:43:05.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:43:05.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:43:05.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:43:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:43:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:43:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:43:05.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:43:05.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:05.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:05.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:43:05.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:43:05.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:43:05.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:43:05.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:43:05.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:05.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:05.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:43:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-10T17:43:05.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:43:05.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:43:05.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:43:05.751 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:43:05.751 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:05.751 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:05.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:43:05.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T17:43:05.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:43:05.953 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:43:05.954 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:05.954 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:05.954 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:43:05.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:43:05.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:43:06.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:06.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:43:06.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:43:06.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:43:06.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: test_get_not_primary: run_mgr td/ceph-helpers x 2024-09-10T17:43:06.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:43:06.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:43:06.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:43:06.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:43:06.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:43:06.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:43:06.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:43:06.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:43:06.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:43:06.957 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:43:06.957 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:06.957 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:06.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:43:06.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:43:06.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:43:06.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: test_get_not_primary: run_osd td/ceph-helpers 0 2024-09-10T17:43:06.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:43:06.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:43:06.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:43:06.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:43:06.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:43:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:43:06.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:43:06.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:43:06.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:43:06.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:43:06.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6b50d5ee-c810-4212-bd31-66fee801f1a8 2024-09-10T17:43:06.991 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 6b50d5ee-c810-4212-bd31-66fee801f1a8 2024-09-10T17:43:06.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6b50d5ee-c810-4212-bd31-66fee801f1a8' 2024-09-10T17:43:06.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:43:07.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQArheBmP7dpABAA+0hv2QXt8VuhP6PQI4f2Bw== 2024-09-10T17:43:07.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQArheBmP7dpABAA+0hv2QXt8VuhP6PQI4f2Bw=="}' 2024-09-10T17:43:07.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6b50d5ee-c810-4212-bd31-66fee801f1a8 -i td/ceph-helpers/0/new.json 2024-09-10T17:43:07.276 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:43:07.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:43:07.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArheBmP7dpABAA+0hv2QXt8VuhP6PQI4f2Bw== --osd-uuid 6b50d5ee-c810-4212-bd31-66fee801f1a8 2024-09-10T17:43:07.317 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:07.315+0000 7fa9e55dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:07.346 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:07.345+0000 7fa9e55dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:07.347 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:07.346+0000 7fa9e55dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:07.347 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:07.346+0000 7fa9e55dd640 -1 bdev(0x55f77d80fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:43:07.347 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:07.346+0000 7fa9e55dd640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:43:12.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:43:12.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:43:12.936 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:43:12.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:43:12.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:43:13.484 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:43:13.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:43:13.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:43:13.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:43:13.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:43:13.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:43:13.511 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:13.510+0000 7f3225015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:13.513 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:13.512+0000 7f3225015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:13.514 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:13.513+0000 7f3225015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:13.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:43:13.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:43:13.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:43:13.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:43:13.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:43:13.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:13.913 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:43:13.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:43:13.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:13.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:15.113 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:15.112+0000 7f3225015640 -1 Falling back to public interface 2024-09-10T17:43:15.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:15.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:15.331 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:43:15.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:43:15.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:15.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:15.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:16.476 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:16.475+0000 7f3225015640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:43:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:43:16.774 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:43:16.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:16.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:17.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:18.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:18.220 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:43:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:43:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:18.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:19.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:19.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:19.656 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:43:19.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:43:19.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:19.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:20.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:21.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:21.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:21.055 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:43:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:43:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:21.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:22.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:22.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:22.604 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:43:22.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:43:22.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:22.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:23.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:24.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:24.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:24.002 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:43:24.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:43:24.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:24.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:24.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:25.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:25.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:25.420 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:43:25.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:43:25.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:25.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:25.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:26.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:26.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:26.839 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:43:26.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:43:26.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:26.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:27.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:28.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:28.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:28.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:43:28.272 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:43:28.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:28.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:28.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:29.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:29.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:29.891 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:43:29.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:43:29.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:29.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:30.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:31.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:31.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:31.440 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:43:31.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:43:31.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:31.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:32.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:33.110 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:43:33.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:43:33.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:33.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:34.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:35.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:35.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:35.275 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:43:35.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:43:35.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:35.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:36.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:37.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:37.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:37.084 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:43:37.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:43:37.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:37.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:38.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:39.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:39.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:39.019 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:43:39.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:43:39.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:39.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:43:39.572 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/677120312,v1:127.0.0.1:6803/677120312] [v2:127.0.0.1:6804/677120312,v1:127.0.0.1:6805/677120312] exists,up 6b50d5ee-c810-4212-bd31-66fee801f1a8 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: test_get_not_primary: run_osd td/ceph-helpers 1 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:43:39.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2024-09-10T17:43:39.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:43:39.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:43:39.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:43:39.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:43:39.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-10T17:43:39.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-10T17:43:39.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:43:39.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:43:39.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:43:39.577 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:43:39.577 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:43:39.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:43:39.578 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:43:39.578 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:39.578 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:43:39.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:43:39.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:43:39.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:43:39.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:43:39.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:43:39.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:43:39.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:43:39.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:43:39.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:43:39.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:43:39.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:43:39.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:43:39.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:43:39.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2024-09-10T17:43:39.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:43:39.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f0d925f6-7f33-4d6a-aba0-c099f7275b5d 2024-09-10T17:43:39.581 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 f0d925f6-7f33-4d6a-aba0-c099f7275b5d 2024-09-10T17:43:39.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f0d925f6-7f33-4d6a-aba0-c099f7275b5d' 2024-09-10T17:43:39.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:43:39.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBLheBmI8ZlIxAA8emWlyL0Ry6rt48HhHx2YA== 2024-09-10T17:43:39.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBLheBmI8ZlIxAA8emWlyL0Ry6rt48HhHx2YA=="}' 2024-09-10T17:43:39.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f0d925f6-7f33-4d6a-aba0-c099f7275b5d -i td/ceph-helpers/1/new.json 2024-09-10T17:43:40.113 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:43:40.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2024-09-10T17:43:40.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLheBmI8ZlIxAA8emWlyL0Ry6rt48HhHx2YA== --osd-uuid f0d925f6-7f33-4d6a-aba0-c099f7275b5d 2024-09-10T17:43:40.155 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:40.153+0000 7f127e81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:40.166 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:40.164+0000 7f127e81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:40.166 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:40.165+0000 7f127e81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:40.166 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:40.165+0000 7f127e81e640 -1 bdev(0x557573001c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:43:40.166 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:40.165+0000 7f127e81e640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-10T17:43:46.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-10T17:43:46.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:43:46.256 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-10T17:43:46.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:43:46.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:43:46.787 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-10T17:43:46.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:43:46.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:43:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:43:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:43:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:43:46.814 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:46.813+0000 7fbb329c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:46.842 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:46.841+0000 7fbb329c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:46.843 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:46.842+0000 7fbb329c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:47.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:43:47.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:43:47.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:43:47.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:43:47.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:43:47.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:47.299 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:43:47.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:43:47.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:47.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:47.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:48.682 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:48.681+0000 7fbb329c5640 -1 Falling back to public interface 2024-09-10T17:43:48.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:48.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:48.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:43:48.717 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:43:48.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:48.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:49.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:50.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:50.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:50.144 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:43:50.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:43:50.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:50.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:50.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:50.595 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:43:50.594+0000 7fbb329c5640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:43:51.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:51.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:51.528 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:43:51.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:43:51.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:51.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:51.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:52.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:52.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:52.949 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:43:52.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:43:52.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:52.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:53.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:54.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:54.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:54.400 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:43:54.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:43:54.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:54.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:54.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:55.814 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:43:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:43:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:56.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:57.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:57.252 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:43:57.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:57.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:43:57.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:57.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:57.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:58.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:58.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:58.734 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:43:58.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:43:58.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:58.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:43:59.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:00.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:00.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:00.205 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:44:00.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:44:00.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:00.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:00.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:01.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:01.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:01.605 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:44:01.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:44:01.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:01.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:02.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:03.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:03.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:03.003 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:44:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:44:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:03.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:04.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:04.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:04.569 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:44:04.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:44:04.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:04.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:05.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:06.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:06.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:06.196 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:44:06.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:44:06.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:06.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:06.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:07.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:07.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:07.895 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:44:07.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:44:07.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:07.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:08.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:09.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:09.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:09.686 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:44:09.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:44:09.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:09.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:10.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:11.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:11.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:11.819 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:44:11.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:44:11.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:11.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:12.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:13.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:13.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:13.336 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:44:13.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:44:13.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:13.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:13.771 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/535075347,v1:127.0.0.1:6811/535075347] [v2:127.0.0.1:6812/535075347,v1:127.0.0.1:6813/535075347] exists,up f0d925f6-7f33-4d6a-aba0-c099f7275b5d 2024-09-10T17:44:13.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:44:13.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:44:13.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:44:13.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: test_get_not_primary: create_rbd_pool 2024-09-10T17:44:13.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:44:14.186 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:44:14.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:44:14.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:44:14.679 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:44:14.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:44:15.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:44:17.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: test_get_not_primary: wait_for_clean 2024-09-10T17:44:17.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:44:17.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:44:17.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:44:17.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:44:17.618 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:44:17.618 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:44:17.618 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:44:17.618 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:44:17.618 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:44:17.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:44:17.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:44:17.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:44:17.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:44:17.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:44:17.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:44:18.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:44:18.133 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2024-09-10T17:44:18.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:44:18.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:44:18.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:44:18.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-10T17:44:18.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-10T17:44:18.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-10T17:44:18.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:44:18.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:44:18.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672966 2024-09-10T17:44:18.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672966 2024-09-10T17:44:18.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492 1-42949672966' 2024-09-10T17:44:18.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:44:18.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-10T17:44:18.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:44:18.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:44:18.365 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-10T17:44:18.365 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:44:18.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-10T17:44:18.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-10T17:44:18.366 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836492 2024-09-10T17:44:18.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:44:18.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-10T17:44:18.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:44:19.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:44:19.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:44:20.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-10T17:44:20.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:44:20.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672966 2024-09-10T17:44:20.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:44:20.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:44:20.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672966 2024-09-10T17:44:20.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:44:20.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672966 2024-09-10T17:44:20.214 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672966 2024-09-10T17:44:20.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-10T17:44:20.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:44:20.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-10T17:44:20.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:44:20.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:44:20.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:44:21.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:44:21.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:44:21.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:44:21.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:44:21.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:44:21.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:44:21.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:44:21.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:44:21.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:44:21.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:44:21.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:44:21.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:44:22.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:44:22.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:44:22.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:44:22.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: get_primary rbd GROUP 2024-09-10T17:44:22.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-10T17:44:22.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2024-09-10T17:44:22.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2024-09-10T17:44:22.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T17:44:22.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local primary=1 2024-09-10T17:44:22.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: get_not_primary rbd GROUP 2024-09-10T17:44:22.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-10T17:44:22.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=GROUP 2024-09-10T17:44:22.603 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd GROUP 2024-09-10T17:44:22.603 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-10T17:44:22.603 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2024-09-10T17:44:22.603 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2024-09-10T17:44:22.603 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T17:44:23.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T17:44:23.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd GROUP 2024-09-10T17:44:23.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T17:44:23.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: local not_primary=0 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: test 0 '!=' 1 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: teardown td/ceph-helpers 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:44:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:44:23.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:44:23.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:44:23.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:44:23.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:44:23.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:44:23.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:44:23.598 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:44:23.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:44:23.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:44:23.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:44:23.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:44:23.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:44:23.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:44:23.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:44:23.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:44:23.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:23.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:44:23.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:44:23.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:44:23.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_num_active_clean td/ceph-helpers 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: test_get_num_active_clean: local dir=td/ceph-helpers 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: test_get_num_active_clean: setup td/ceph-helpers 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:44:23.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:44:23.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:44:23.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:44:23.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:44:23.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:44:23.614 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:44:23.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:44:23.615 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:44:23.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:44:23.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:44:23.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:44:23.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:44:23.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:44:23.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:44:23.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:44:23.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:44:23.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:44:23.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:44:23.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:44:23.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:23.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:44:23.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:44:23.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:44:23.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:44:23.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:44:23.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:44:23.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:23.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:44:23.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:44:23.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:44:23.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:44:23.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:44:23.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:44:23.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:44:23.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:44:23.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:44:23.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:44:23.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:44:23.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:44:23.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:44:24.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:44:24.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:44:24.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:44:24.091 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:44:24.091 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:24.091 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:44:24.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:44:24.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:44:24.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:44:24.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:44:24.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:44:24.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:44:24.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:44:24.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:44:24.285 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:44:24.285 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:44:24.286 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:44:24.286 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:24.286 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:24.286 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:44:24.286 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:44:24.286 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:44:24.286 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:44:24.367 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:44:24.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:44:24.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:44:24.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-09-10T17:44:24.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:44:24.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:44:24.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:44:24.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:44:24.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:44:24.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:44:25.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:44:25.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:44:25.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:44:25.297 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:44:25.297 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:25.298 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:44:25.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:44:25.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:44:25.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:44:25.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:44:25.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:44:25.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:44:25.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=17e6eb5d-2d7a-40a8-9d40-5cb7def15d49 2024-09-10T17:44:25.334 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 17e6eb5d-2d7a-40a8-9d40-5cb7def15d49 2024-09-10T17:44:25.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 17e6eb5d-2d7a-40a8-9d40-5cb7def15d49' 2024-09-10T17:44:25.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:44:25.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB5heBm/UTUFBAAntreD1pYUW721wsHziT5iw== 2024-09-10T17:44:25.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB5heBm/UTUFBAAntreD1pYUW721wsHziT5iw=="}' 2024-09-10T17:44:25.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 17e6eb5d-2d7a-40a8-9d40-5cb7def15d49 -i td/ceph-helpers/0/new.json 2024-09-10T17:44:25.616 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:44:25.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:44:25.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5heBm/UTUFBAAntreD1pYUW721wsHziT5iw== --osd-uuid 17e6eb5d-2d7a-40a8-9d40-5cb7def15d49 2024-09-10T17:44:25.656 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:25.655+0000 7efcb0228640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:25.685 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:25.684+0000 7efcb0228640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:25.687 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:25.686+0000 7efcb0228640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:25.687 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:25.686+0000 7efcb0228640 -1 bdev(0x55c433705c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:44:25.687 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:25.686+0000 7efcb0228640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:44:31.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:44:31.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:44:31.700 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:44:31.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:44:31.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:44:32.239 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:44:32.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:44:32.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:44:32.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:44:32.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:44:32.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:44:32.266 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:32.265+0000 7fb7e725e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:32.284 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:32.283+0000 7fb7e725e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:32.285 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:32.284+0000 7fb7e725e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:32.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:44:32.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:44:32.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:44:32.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:44:32.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:44:32.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:32.693 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:44:32.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:44:32.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:32.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:33.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:33.382 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:33.382+0000 7fb7e725e640 -1 Falling back to public interface 2024-09-10T17:44:34.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:34.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:34.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:44:34.117 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:44:34.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:34.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:34.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:35.446 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:44:35.444+0000 7fb7e725e640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:44:35.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:35.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:35.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:44:35.511 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:44:35.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:35.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:36.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:37.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:37.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:37.065 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:44:37.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:44:37.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:37.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:37.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:38.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:38.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:38.497 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:44:38.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:44:38.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:38.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:38.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:39.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:39.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:39.906 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:44:39.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:44:39.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:39.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:40.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:41.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:41.307 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:44:41.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:41.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:44:41.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:41.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:42.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:42.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:42.738 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:44:42.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:44:42.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:42.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:43.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:44.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:44.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:44.218 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:44:44.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:44:44.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:44.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:44.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:45.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:45.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:45.616 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:44:45.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:44:45.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:45.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:46.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:47.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:47.021 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:44:47.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:47.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:44:47.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:47.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:48.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:48.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:48.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:44:48.796 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:44:48.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:48.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:49.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:50.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:50.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:50.433 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:44:50.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:44:50.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:50.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:51.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:52.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:52.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:52.627 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:44:52.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:44:52.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:52.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:53.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:54.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:54.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:54.033 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:44:54.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:44:54.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:54.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:55.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:56.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:56.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:56.009 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:44:56.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:44:56.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:56.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:56.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:57.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:57.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:57.873 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:44:57.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:44:57.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:57.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:58.272 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1520420875,v1:127.0.0.1:6803/1520420875] [v2:127.0.0.1:6804/1520420875,v1:127.0.0.1:6805/1520420875] exists,up 17e6eb5d-2d7a-40a8-9d40-5cb7def15d49 2024-09-10T17:44:58.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:44:58.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:44:58.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:44:58.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1369: test_get_num_active_clean: create_rbd_pool 2024-09-10T17:44:58.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:44:58.704 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:44:58.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:44:58.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:44:59.308 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:44:59.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:45:00.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:45:01.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1370: test_get_num_active_clean: wait_for_clean 2024-09-10T17:45:01.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:45:01.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:45:01.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:45:01.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:45:01.408 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:45:01.409 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:45:01.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:45:01.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:45:01.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:45:01.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:45:01.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:45:01.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:45:01.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:45:01.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:45:01.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:45:01.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:45:01.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:45:01.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:45:01.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:45:02.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836486 2024-09-10T17:45:02.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836486 2024-09-10T17:45:02.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836486' 2024-09-10T17:45:02.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:45:02.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836486 2024-09-10T17:45:02.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:45:02.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:45:02.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836486 2024-09-10T17:45:02.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:45:02.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836486 2024-09-10T17:45:02.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-10T17:45:02.026 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-10T17:45:02.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:45:02.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-10T17:45:02.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:45:03.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:45:03.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:45:03.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-10T17:45:03.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:45:03.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:45:03.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:45:04.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:45:04.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:45:04.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:45:04.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:45:04.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:45:04.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:45:04.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:45:04.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:45:04.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:45:04.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:45:04.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:45:04.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: get_num_active_clean 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:45:05.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:45:05.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: local num_active_clean=4 2024-09-10T17:45:05.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: test 4 = 4 2024-09-10T17:45:05.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1373: test_get_num_active_clean: teardown td/ceph-helpers 2024-09-10T17:45:05.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:45:05.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:45:05.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:45:05.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:45:05.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:45:05.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:45:05.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:45:05.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:45:05.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:45:05.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:45:05.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:45:05.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:45:05.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:45:05.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:45:05.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:45:05.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:45:05.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:45:05.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:45:05.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:45:05.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:45:05.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:45:05.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:45:05.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:45:05.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:05.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:05.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:45:05.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:45:05.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1393: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: test_get_num_active_or_peered: setup td/ceph-helpers 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:45:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:45:05.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:45:05.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:45:05.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:45:05.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:45:05.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:45:05.855 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:45:05.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:45:05.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:45:05.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:45:05.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:45:05.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:45:05.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:45:05.857 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:45:05.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:45:05.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:45:05.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:45:05.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:45:05.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:05.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:05.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:45:05.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:45:05.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:45:05.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:45:05.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:45:05.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:05.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:05.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:45:05.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:45:06.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:45:06.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:45:06.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:45:06.320 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:45:06.320 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:06.320 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:06.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:45:06.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:45:06.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:45:06.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:45:06.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:45:06.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:45:06.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:45:06.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:45:06.538 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:45:06.538 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:45:06.538 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:45:06.539 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:45:06.539 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:06.539 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:06.539 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:45:06.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:45:06.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:45:06.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:06.621 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:45:06.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:45:06.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:45:06.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-09-10T17:45:06.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:45:06.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:45:06.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:45:06.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:45:06.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:45:06.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:45:07.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:45:07.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:45:07.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:45:07.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:45:07.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:07.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:07.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:45:07.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:45:07.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:45:07.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-09-10T17:45:07.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:45:07.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:45:07.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:45:07.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:45:07.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:45:07.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:45:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:45:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:45:07.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:45:07.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:45:07.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:45:07.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0c7fde34-796a-4e46-bca0-a93e4a225849 2024-09-10T17:45:07.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0c7fde34-796a-4e46-bca0-a93e4a225849' 2024-09-10T17:45:07.586 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 0c7fde34-796a-4e46-bca0-a93e4a225849 2024-09-10T17:45:07.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:45:07.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCjheBmjf/SIxAAQlFqSiwxiJru03WBGGKSIQ== 2024-09-10T17:45:07.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCjheBmjf/SIxAAQlFqSiwxiJru03WBGGKSIQ=="}' 2024-09-10T17:45:07.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0c7fde34-796a-4e46-bca0-a93e4a225849 -i td/ceph-helpers/0/new.json 2024-09-10T17:45:07.878 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:45:07.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:45:07.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjheBmjf/SIxAAQlFqSiwxiJru03WBGGKSIQ== --osd-uuid 0c7fde34-796a-4e46-bca0-a93e4a225849 2024-09-10T17:45:07.918 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:07.917+0000 7fbce2bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:07.947 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:07.946+0000 7fbce2bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:07.948 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:07.947+0000 7fbce2bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:07.949 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:07.948+0000 7fbce2bcc640 -1 bdev(0x5630cc43fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:45:07.949 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:07.948+0000 7fbce2bcc640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:45:13.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:45:13.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:45:13.645 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:45:13.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:45:13.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:45:14.240 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:45:14.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:45:14.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:45:14.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:45:14.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:45:14.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:45:14.267 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:14.266+0000 7f36c6bc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:14.270 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:14.268+0000 7f36c6bc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:14.271 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:14.270+0000 7f36c6bc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:14.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:45:14.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:45:14.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:45:14.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:45:14.669 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:45:14.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:45:14.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:14.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:45:14.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:14.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:15.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:15.873 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:15.872+0000 7f36c6bc5640 -1 Falling back to public interface 2024-09-10T17:45:16.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:16.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:16.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:45:16.059 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:45:16.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:16.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:16.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:17.456 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:17.455+0000 7f36c6bc5640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:45:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:17.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:17.487 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:45:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:45:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:18.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:19.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:19.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:19.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:45:19.067 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:45:19.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:19.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:19.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:20.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:20.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:20.514 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:45:20.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:45:20.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:20.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:20.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:21.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:21.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:21.913 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:45:21.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:45:21.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:21.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:22.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:23.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:23.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:23.360 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:45:23.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:45:23.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:23.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:23.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:24.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:24.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:24.774 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:45:24.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:45:24.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:24.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:25.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:26.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:26.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:26.206 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:45:26.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:45:26.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:26.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:26.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:27.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:27.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:27.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:45:27.637 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:45:27.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:27.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:28.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:29.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:29.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:29.039 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:45:29.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:45:29.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:29.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:30.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:31.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:31.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:31.005 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:45:31.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:45:31.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:31.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:31.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:32.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:32.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:32.683 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:45:32.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:45:32.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:32.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:33.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:34.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:34.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:34.394 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:45:34.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:45:34.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:34.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:35.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:36.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:36.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:36.230 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:45:36.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:45:36.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:36.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:37.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:38.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:38.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:38.124 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:45:38.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:45:38.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:38.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:39.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:40.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:40.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:40.279 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:45:40.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:45:40.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:40.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:40.756 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/619097312,v1:127.0.0.1:6803/619097312] [v2:127.0.0.1:6804/619097312,v1:127.0.0.1:6805/619097312] exists,up 0c7fde34-796a-4e46-bca0-a93e4a225849 2024-09-10T17:45:40.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:45:40.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:45:40.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:45:40.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1399: test_get_num_active_or_peered: create_rbd_pool 2024-09-10T17:45:40.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:45:41.139 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:45:41.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:45:41.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:45:41.631 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:45:41.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:45:42.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:45:43.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1400: test_get_num_active_or_peered: wait_for_clean 2024-09-10T17:45:43.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:45:43.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:45:43.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:45:43.736 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:45:43.736 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:45:43.736 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:45:43.736 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:45:43.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:45:43.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:45:43.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:45:43.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:45:43.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:45:43.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:45:43.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:45:43.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:45:44.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:45:44.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:45:44.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:45:44.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:45:44.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-10T17:45:44.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-10T17:45:44.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-10T17:45:44.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:45:44.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-10T17:45:44.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:45:44.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:45:44.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-10T17:45:44.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:45:44.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-10T17:45:44.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-10T17:45:44.369 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836488 2024-09-10T17:45:44.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:45:44.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-10T17:45:44.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:45:45.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:45:45.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:45:46.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-10T17:45:46.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:45:46.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:45:46.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:45:46.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:45:46.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:45:46.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:45:46.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:45:46.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:45:46.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:45:46.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:45:46.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:45:47.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:45:47.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:45:47.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:45:47.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:45:47.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:45:47.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:45:47.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:45:47.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: get_num_active_or_peered 2024-09-10T17:45:47.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2024-09-10T17:45:47.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-10T17:45:47.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-10T17:45:47.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-10T17:45:47.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-10T17:45:48.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: local num_peered=4 2024-09-10T17:45:48.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: test 4 = 4 2024-09-10T17:45:48.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-09-10T17:45:48.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:45:48.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:45:48.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:45:48.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:45:48.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:45:48.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:45:48.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:45:48.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:45:48.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:45:48.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:45:48.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:45:48.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:45:48.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:45:48.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:45:48.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:45:48.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:45:48.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:45:48.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:45:48.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:45:48.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:45:48.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:45:48.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:45:48.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:45:48.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:48.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:48.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:45:48.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:45:48.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:45:48.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_num_pgs td/ceph-helpers 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1420: test_get_num_pgs: local dir=td/ceph-helpers 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1422: test_get_num_pgs: setup td/ceph-helpers 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:45:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:45:48.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:45:48.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:45:48.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:45:48.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:45:48.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:45:48.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:45:48.245 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:45:48.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:45:48.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:45:48.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:45:48.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:45:48.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:45:48.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:45:48.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:45:48.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:45:48.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:48.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:48.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:45:48.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:45:48.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:45:48.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:45:48.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:45:48.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:48.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:48.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:45:48.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:45:48.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:45:48.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:45:48.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:45:48.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:45:48.694 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:45:48.694 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:48.694 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:48.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:45:48.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:45:48.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:45:48.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:45:48.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:45:48.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:45:48.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:45:48.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:45:48.887 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:45:48.887 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:45:48.887 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:45:48.888 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:45:48.888 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:48.888 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:48.888 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:45:48.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:45:48.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:48.969 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:48.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:45:48.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:45:48.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:45:49.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1424: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-09-10T17:45:49.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:45:49.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:45:49.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:45:49.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:45:49.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:45:49.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:45:49.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:45:49.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:45:49.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:45:49.891 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:45:49.891 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:49.891 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:49.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:45:49.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:45:49.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:45:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:45:49.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:45:49.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:45:49.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c3a1225e-c513-426e-994b-8c615511a49c 2024-09-10T17:45:49.925 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 c3a1225e-c513-426e-994b-8c615511a49c 2024-09-10T17:45:49.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c3a1225e-c513-426e-994b-8c615511a49c' 2024-09-10T17:45:49.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:45:49.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNheBmAdkUOBAAZtR0zbUCa46bIyeGYXIeiw== 2024-09-10T17:45:49.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNheBmAdkUOBAAZtR0zbUCa46bIyeGYXIeiw=="}' 2024-09-10T17:45:49.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c3a1225e-c513-426e-994b-8c615511a49c -i td/ceph-helpers/0/new.json 2024-09-10T17:45:50.202 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:45:50.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:45:50.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNheBmAdkUOBAAZtR0zbUCa46bIyeGYXIeiw== --osd-uuid c3a1225e-c513-426e-994b-8c615511a49c 2024-09-10T17:45:50.242 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:50.241+0000 7fa92a80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:50.273 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:50.272+0000 7fa92a80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:50.274 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:50.273+0000 7fa92a80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:50.274 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:50.273+0000 7fa92a80d640 -1 bdev(0x55860bb1dc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:45:50.275 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:50.273+0000 7fa92a80d640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:45:55.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:45:55.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:45:55.762 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:45:55.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:45:55.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:45:56.318 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:45:56.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:45:56.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:45:56.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:45:56.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:45:56.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:45:56.345 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:56.344+0000 7ff5aa0f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:56.347 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:56.346+0000 7ff5aa0f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:56.348 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:56.347+0000 7ff5aa0f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:56.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:45:56.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:45:56.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:45:56.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:45:56.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:45:56.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:56.753 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:45:56.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:45:56.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:56.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:57.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:57.693 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:57.692+0000 7ff5aa0f1640 -1 Falling back to public interface 2024-09-10T17:45:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:58.135 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:45:58.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:58.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:45:58.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:58.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:45:58.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:59.265 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:45:59.264+0000 7ff5aa0f1640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:45:59.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:59.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:59.564 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:45:59.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:45:59.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:59.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:00.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:01.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:01.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:01.026 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:46:01.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:46:01.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:01.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:01.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:02.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:02.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:02.422 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:46:02.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:46:02.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:02.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:02.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:03.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:03.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:03.825 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:46:03.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:46:03.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:03.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:05.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:05.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:05.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:46:05.227 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:46:05.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:05.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:05.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:06.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:06.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:06.644 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:46:06.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:46:06.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:06.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:07.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:08.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:08.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:08.201 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:46:08.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:46:08.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:08.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:08.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:09.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:09.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:09.597 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:46:09.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:46:09.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:09.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:09.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:10.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:10.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:10.994 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:46:10.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:46:10.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:10.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:11.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:12.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:12.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:12.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:46:12.920 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:46:12.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:12.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:13.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:14.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:14.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:14.479 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:46:14.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:46:14.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:14.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:15.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:16.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:16.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:16.214 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:46:16.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:46:16.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:16.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:16.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:17.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:17.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:17.900 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:46:17.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:46:17.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:17.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:18.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:19.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:19.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:19.957 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:46:19.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:46:19.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:19.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:20.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:21.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:21.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:21.878 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:46:21.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:46:21.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:21.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:22.328 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1305778783,v1:127.0.0.1:6803/1305778783] [v2:127.0.0.1:6804/1305778783,v1:127.0.0.1:6805/1305778783] exists,up c3a1225e-c513-426e-994b-8c615511a49c 2024-09-10T17:46:22.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:46:22.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:46:22.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:46:22.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1426: test_get_num_pgs: create_rbd_pool 2024-09-10T17:46:22.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:46:22.725 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:46:22.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:46:22.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:46:23.235 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:46:23.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:46:24.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:46:25.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1427: test_get_num_pgs: wait_for_clean 2024-09-10T17:46:25.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:46:25.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:46:25.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:46:25.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:46:25.420 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:46:25.420 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:46:25.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:46:25.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:46:25.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:46:25.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:46:25.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:46:25.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:46:25.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:46:25.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:46:25.515 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:46:25.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:46:25.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:46:25.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:46:25.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:46:26.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-10T17:46:26.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-10T17:46:26.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-10T17:46:26.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:46:26.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-10T17:46:26.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:46:26.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:46:26.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-10T17:46:26.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:46:26.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-10T17:46:26.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-10T17:46:26.020 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836487 2024-09-10T17:46:26.020 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:46:26.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-10T17:46:26.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:46:27.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:46:27.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:46:27.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-10T17:46:27.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:46:28.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T17:46:28.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:46:29.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836487 2024-09-10T17:46:29.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:46:29.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:46:29.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:46:29.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:46:29.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:46:29.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:46:29.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:46:29.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:46:29.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:46:29.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:46:29.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:46:30.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:46:30.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:46:30.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:46:30.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:46:30.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:46:30.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:46:30.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:46:30.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: get_num_pgs 2024-09-10T17:46:30.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:46:30.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:46:31.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: local num_pgs=4 2024-09-10T17:46:31.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: test 4 -gt 0 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1430: test_get_num_pgs: teardown td/ceph-helpers 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:46:31.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:46:31.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:46:31.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:46:31.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:46:31.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:46:31.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:46:31.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:46:31.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:46:31.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:46:31.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:46:31.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:46:31.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:46:31.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:46:31.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:46:31.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:46:31.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:46:31.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:31.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:31.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:46:31.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:46:31.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:46:31.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_op_scheduler td/ceph-helpers 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: test_get_op_scheduler: local dir=td/ceph-helpers 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: test_get_op_scheduler: setup td/ceph-helpers 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:46:31.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:46:31.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:46:31.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:46:31.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:46:31.450 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:46:31.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:46:31.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:46:31.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:46:31.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:46:31.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:46:31.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:46:31.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:46:31.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:46:31.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:46:31.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:46:31.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:46:31.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:31.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:31.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:46:31.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:46:31.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:46:31.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:46:31.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:46:31.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:31.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:31.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:46:31.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:46:31.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-10T17:46:31.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:46:31.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:46:31.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:46:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:46:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:31.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:46:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:46:32.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:46:32.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:46:32.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:46:32.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:46:32.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:46:32.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:46:32.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:46:32.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:46:32.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:46:32.096 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:32.096 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:32.096 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:32.096 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:46:32.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:46:32.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:46:32.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:46:32.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:46:32.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:46:32.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:46:32.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:46:32.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:46:32.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-09-10T17:46:32.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:46:32.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:46:32.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:46:32.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:46:32.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:46:32.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:46:33.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:46:33.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:46:33.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:46:33.099 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:46:33.099 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:33.099 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:33.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:46:33.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:46:33.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:46:33.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-09-10T17:46:33.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:46:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:46:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=wpq 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:46:33.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:46:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ea7bd5fb-f7be-48c3-a1f2-66da24661ba0 2024-09-10T17:46:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ea7bd5fb-f7be-48c3-a1f2-66da24661ba0' 2024-09-10T17:46:33.134 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 ea7bd5fb-f7be-48c3-a1f2-66da24661ba0 2024-09-10T17:46:33.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:46:33.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD5heBmL8vkCBAA7g+nQT1AsZAFWcpLZiRfnQ== 2024-09-10T17:46:33.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD5heBmL8vkCBAA7g+nQT1AsZAFWcpLZiRfnQ=="}' 2024-09-10T17:46:33.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ea7bd5fb-f7be-48c3-a1f2-66da24661ba0 -i td/ceph-helpers/0/new.json 2024-09-10T17:46:33.427 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:46:33.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:46:33.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQD5heBmL8vkCBAA7g+nQT1AsZAFWcpLZiRfnQ== --osd-uuid ea7bd5fb-f7be-48c3-a1f2-66da24661ba0 2024-09-10T17:46:33.467 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:33.466+0000 7f5016c2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:33.496 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:33.495+0000 7f5016c2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:33.497 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:33.496+0000 7f5016c2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:33.497 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:33.496+0000 7f5016c2a640 -1 bdev(0x55a00fde9c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:46:33.497 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:33.496+0000 7f5016c2a640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:46:39.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:46:39.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:46:39.001 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:46:39.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:46:39.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:46:39.569 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:46:39.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:46:39.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-09-10T17:46:39.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:46:39.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:46:39.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:46:39.596 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:39.595+0000 7f130a49c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:39.610 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:39.608+0000 7f130a49c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:39.611 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:39.610+0000 7f130a49c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:40.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:46:40.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:46:40.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:46:40.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:46:40.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:46:40.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:40.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:46:40.008 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:46:40.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:40.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:40.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:40.966 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:40.965+0000 7f130a49c640 -1 Falling back to public interface 2024-09-10T17:46:41.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:41.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:41.434 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:46:41.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:46:41.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:41.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:41.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:42.546 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:42.545+0000 7f130a49c640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:46:42.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:42.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:42.869 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:46:42.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:46:42.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:42.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:43.331 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/412783017,v1:127.0.0.1:6803/412783017] [v2:127.0.0.1:6804/412783017,v1:127.0.0.1:6805/412783017] exists,up ea7bd5fb-f7be-48c3-a1f2-66da24661ba0 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: test_get_op_scheduler: get_op_scheduler 0 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: get_op_scheduler: local id=0 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: get_op_scheduler: get_config osd 0 osd_op_queue 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2024-09-10T17:46:43.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.0.asok 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:46:43.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-osd.0.asok config get osd_op_queue 2024-09-10T17:46:43.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: test_get_op_scheduler: test wpq = wpq 2024-09-10T17:46:43.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-09-10T17:46:43.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:46:43.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:46:43.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:46:43.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:46:43.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:46:43.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:46:43.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:46:43.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:46:43.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-10T17:46:43.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2024-09-10T17:46:43.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:46:43.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0e7f3210-3a48-4728-afec-d488e43d05c9 2024-09-10T17:46:43.425 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 0e7f3210-3a48-4728-afec-d488e43d05c9 2024-09-10T17:46:43.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 0e7f3210-3a48-4728-afec-d488e43d05c9' 2024-09-10T17:46:43.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:46:43.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQADhuBmpNksGhAA0HKmVIDFcslGt3c6RD1OSA== 2024-09-10T17:46:43.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQADhuBmpNksGhAA0HKmVIDFcslGt3c6RD1OSA=="}' 2024-09-10T17:46:43.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0e7f3210-3a48-4728-afec-d488e43d05c9 -i td/ceph-helpers/1/new.json 2024-09-10T17:46:43.925 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:46:43.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2024-09-10T17:46:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQADhuBmpNksGhAA0HKmVIDFcslGt3c6RD1OSA== --osd-uuid 0e7f3210-3a48-4728-afec-d488e43d05c9 2024-09-10T17:46:43.966 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:43.965+0000 7fd27a145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:43.999 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:43.998+0000 7fd27a145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:44.000 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:43.999+0000 7fd27a145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:44.000 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:43.999+0000 7fd27a145640 -1 bdev(0x558cf6291c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:46:44.000 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:43.999+0000 7fd27a145640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-10T17:46:49.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-10T17:46:49.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:46:49.949 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-10T17:46:49.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:46:49.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:46:50.487 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-10T17:46:50.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:46:50.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-10T17:46:50.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:46:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:46:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:46:50.517 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:50.515+0000 7f4ed92f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:50.523 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:50.521+0000 7f4ed92f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:50.524 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:50.523+0000 7f4ed92f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:50.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:46:50.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:46:50.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:46:50.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:46:50.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:46:50.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:50.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:46:50.937 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:46:50.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:50.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:46:51.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:51.871 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:51.870+0000 7f4ed92f1640 -1 Falling back to public interface 2024-09-10T17:46:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:52.372 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:46:52.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:46:52.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:52.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:46:52.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:53.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:53.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:53.800 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:46:53.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:46:53.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:53.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:46:53.860 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:46:53.859+0000 7f4ed92f1640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:46:54.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:55.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:55.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:55.203 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:46:55.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:46:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:46:55.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:46:56.679 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:46:56.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:56.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:46:57.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:58.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:58.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:58.100 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:46:58.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:46:58.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:58.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:46:58.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:59.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:59.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:59.562 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:46:59.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:46:59.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:59.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:00.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:01.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:01.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:01.002 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:47:01.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:47:01.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:01.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:01.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:02.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:02.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:02.427 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:47:02.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:47:02.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:02.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:02.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:47:03.863 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:47:03.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:03.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:04.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:05.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:05.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:05.263 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:47:05.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:47:05.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:05.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:06.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:07.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:07.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:07.102 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:47:07.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:47:07.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:07.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:08.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:09.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:09.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:09.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:47:09.013 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:47:09.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:09.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:09.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:10.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:10.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:10.738 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:47:10.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:47:10.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:10.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:11.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:12.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:12.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:12.525 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:47:12.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:47:12.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:12.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:13.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:14.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:14.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:14.361 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:47:14.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:47:14.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:14.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:15.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:16.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:16.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:16.492 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:47:16.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:47:16.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:16.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:16.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:17.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:17.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:17.894 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:47:17.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:47:17.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:17.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:18.347 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3806631050,v1:127.0.0.1:6811/3806631050] [v2:127.0.0.1:6812/3806631050,v1:127.0.0.1:6813/3806631050] exists,up 0e7f3210-3a48-4728-afec-d488e43d05c9 2024-09-10T17:47:18.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:47:18.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:47:18.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:47:18.349 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: test_get_op_scheduler: get_op_scheduler 1 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: get_op_scheduler: local id=1 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: get_op_scheduler: get_config osd 1 osd_op_queue 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=1 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.1 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:18.350 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:18.351 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-osd.1.asok 2024-09-10T17:47:18.351 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:47:18.351 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-osd.1.asok config get osd_op_queue 2024-09-10T17:47:18.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-09-10T17:47:18.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: test_get_op_scheduler: teardown td/ceph-helpers 2024-09-10T17:47:18.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:47:18.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:47:18.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:47:18.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:47:18.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:47:18.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:47:18.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:47:18.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:47:18.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:47:18.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:47:18.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:47:18.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:47:18.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:47:18.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:47:18.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:47:18.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:47:18.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:47:18.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:47:18.544 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:47:18.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:47:18.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:47:18.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:47:18.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:47:18.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:18.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:18.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1450: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1452: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:47:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:47:18.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:47:18.555 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:47:18.555 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:47:18.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:47:18.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:47:18.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:47:18.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:47:18.555 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:47:18.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:47:18.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:47:18.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:47:18.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:47:18.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:47:18.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:47:18.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:47:18.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:47:18.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:47:18.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:47:18.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:47:18.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:47:18.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:47:18.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:18.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:18.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:47:18.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:47:18.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:47:18.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:47:18.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:47:18.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:18.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:18.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1453: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:47:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:47:19.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:47:19.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:47:19.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:47:19.026 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:47:19.026 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:19.026 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:19.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:47:19.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:47:19.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:47:19.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:47:19.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:47:19.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:47:19.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:47:19.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:47:19.211 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:47:19.212 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:47:19.212 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:47:19.212 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:47:19.212 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:19.212 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:19.212 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:47:19.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:47:19.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:19.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:47:19.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:47:19.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:47:19.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1454: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-10T17:47:19.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:47:19.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:47:19.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:47:19.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:47:19.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:47:19.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:47:20.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:47:20.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:47:20.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:47:20.223 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:47:20.223 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:20.223 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:20.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:47:20.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:47:20.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:47:20.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:47:20.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:47:20.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:47:20.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:47:20.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:47:20.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:47:20.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e8e5002f-2436-43e0-8f38-173a75038ea8 2024-09-10T17:47:20.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e8e5002f-2436-43e0-8f38-173a75038ea8' 2024-09-10T17:47:20.259 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 e8e5002f-2436-43e0-8f38-173a75038ea8 2024-09-10T17:47:20.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:47:20.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAohuBmgDVQEBAAATo3q25gCOqs0xGY28ORWw== 2024-09-10T17:47:20.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAohuBmgDVQEBAAATo3q25gCOqs0xGY28ORWw=="}' 2024-09-10T17:47:20.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e8e5002f-2436-43e0-8f38-173a75038ea8 -i td/ceph-helpers/0/new.json 2024-09-10T17:47:20.567 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:47:20.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:47:20.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAohuBmgDVQEBAAATo3q25gCOqs0xGY28ORWw== --osd-uuid e8e5002f-2436-43e0-8f38-173a75038ea8 2024-09-10T17:47:20.607 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:20.606+0000 7f753601b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:20.636 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:20.635+0000 7f753601b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:20.637 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:20.636+0000 7f753601b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:20.637 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:20.636+0000 7f753601b640 -1 bdev(0x56146952dc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:47:20.638 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:20.637+0000 7f753601b640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:47:26.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:47:26.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:47:26.174 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:47:26.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:47:26.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:47:26.724 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:47:26.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:47:26.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:47:26.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:47:26.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:47:26.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:47:26.751 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:26.750+0000 7f42c1e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:26.753 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:26.752+0000 7f42c1e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:26.754 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:26.753+0000 7f42c1e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:27.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:47:27.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:47:27.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:47:27.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:47:27.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:47:27.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:27.146 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:47:27.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:47:27.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:27.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:27.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:27.851 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:27.850+0000 7f42c1e44640 -1 Falling back to public interface 2024-09-10T17:47:28.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:28.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:28.573 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:47:28.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:47:28.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:28.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:28.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:29.917 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:47:29.916+0000 7f42c1e44640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:47:29.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:29.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:29.960 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:47:29.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:47:29.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:29.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:30.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:31.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:31.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:31.513 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:47:31.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:47:31.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:31.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:31.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:32.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:32.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:32.975 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:47:32.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:47:32.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:32.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:33.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:34.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:34.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:34.372 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:47:34.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:47:34.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:34.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:34.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:35.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:35.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:35.820 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:47:35.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:47:35.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:35.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:36.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:37.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:37.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:37.253 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:47:37.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:47:37.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:37.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:37.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:38.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:38.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:38.716 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:47:38.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:47:38.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:38.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:39.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:40.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:40.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:40.114 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:47:40.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:47:40.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:40.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:40.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:41.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:41.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:41.865 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:47:41.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:47:41.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:41.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:42.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:43.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:43.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:43.689 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:47:43.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:47:43.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:43.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:44.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:45.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:45.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:45.211 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:47:45.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:47:45.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:45.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:45.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:46.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:46.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:46.894 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:47:46.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:47:46.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:46.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:48.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:49.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:49.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:49.050 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:47:49.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:47:49.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:49.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:49.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:50.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:50.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:50.536 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:47:50.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:47:50.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:50.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:51.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:52.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:52.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:52.457 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:47:52.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:47:52.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:52.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:52.851 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2248509020,v1:127.0.0.1:6803/2248509020] [v2:127.0.0.1:6804/2248509020,v1:127.0.0.1:6805/2248509020] exists,up e8e5002f-2436-43e0-8f38-173a75038ea8 2024-09-10T17:47:52.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:47:52.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:47:52.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:47:52.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1456: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-09-10T17:47:52.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:47:53.236 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:47:53.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:47:53.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:47:53.728 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:47:53.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:47:54.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:47:55.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1457: test_get_osd_id_used_by_pgs: wait_for_clean 2024-09-10T17:47:55.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:47:55.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:47:55.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:47:55.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:47:55.996 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:47:55.996 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:47:55.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:47:55.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:47:55.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:47:56.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:47:56.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:47:56.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:47:56.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:47:56.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:47:56.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:47:56.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:47:56.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:47:56.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:47:56.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:47:56.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836484 2024-09-10T17:47:56.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836484 2024-09-10T17:47:56.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836484' 2024-09-10T17:47:56.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:47:56.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836484 2024-09-10T17:47:56.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:47:56.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:47:56.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836484 2024-09-10T17:47:56.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:47:56.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836484 2024-09-10T17:47:56.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-10T17:47:56.595 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836484 2024-09-10T17:47:56.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:47:57.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-10T17:47:57.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:47:58.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:47:58.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:47:58.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-10T17:47:58.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:47:59.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T17:47:59.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:47:59.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-10T17:47:59.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:47:59.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:47:59.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:48:00.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:48:00.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:48:00.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:48:00.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:48:00.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:48:00.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:48:00.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:48:00.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:48:00.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:48:00.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:48:00.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:48:00.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:48:01.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:48:01.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:48:01.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:48:01.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-10T17:48:01.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: uniq 2024-09-10T17:48:01.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-10T17:48:01.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-10T17:48:01.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: sort 2024-09-10T17:48:01.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-09-10T17:48:01.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: test 0 = 0 2024-09-10T17:48:01.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1460: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-10T17:48:01.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:48:01.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:48:01.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:48:01.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:48:01.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:48:01.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:48:01.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:48:01.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:48:01.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:48:01.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:48:01.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:48:01.943 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:48:01.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:48:01.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:48:01.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:48:01.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:48:01.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:48:01.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:48:01.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:48:01.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:48:01.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:48:01.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:48:01.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:48:01.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:01.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:01.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_osds td/ceph-helpers 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: test_get_osds: local dir=td/ceph-helpers 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1024: test_get_osds: setup td/ceph-helpers 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:48:01.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:48:01.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:48:01.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:48:01.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:48:01.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:48:01.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:48:01.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:48:01.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:48:01.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:48:01.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:48:01.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:48:01.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:48:01.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:48:01.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:48:01.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:48:01.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:48:01.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:01.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:01.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:48:01.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:48:01.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:48:01.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:48:01.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:48:01.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:01.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:01.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:48:01.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:48:01.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:48:01.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-10T17:48:02.400 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:48:02.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:48:02.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:48:02.401 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:48:02.401 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:02.401 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:02.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:48:02.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T17:48:02.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:48:02.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:48:02.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:48:02.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:48:02.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:48:02.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:48:02.586 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:48:02.586 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:48:02.586 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:48:02.587 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:48:02.587 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:02.587 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:02.587 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:48:02.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:48:02.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:48:02.666 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:48:02.667 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:02.667 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:02.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:48:02.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:48:02.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:48:02.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1026: test_get_osds: run_mgr td/ceph-helpers x 2024-09-10T17:48:02.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:48:02.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:48:02.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:48:02.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:48:02.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:48:02.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:48:03.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:48:03.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:48:03.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:48:03.600 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:48:03.600 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:03.600 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:03.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:48:03.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:48:03.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:48:03.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: test_get_osds: run_osd td/ceph-helpers 0 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:48:03.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:48:03.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:48:03.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:48:03.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:48:03.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:48:03.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:48:03.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:48:03.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:48:03.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:48:03.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:48:03.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:48:03.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:48:03.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:48:03.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:48:03.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:48:03.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69658764-ae7c-4ad3-9db0-3aaa3c2162dc 2024-09-10T17:48:03.636 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 69658764-ae7c-4ad3-9db0-3aaa3c2162dc 2024-09-10T17:48:03.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 69658764-ae7c-4ad3-9db0-3aaa3c2162dc' 2024-09-10T17:48:03.637 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:48:03.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBThuBmfMfFJhAAOe1CTS5SFsOwQ5TGQtYimg== 2024-09-10T17:48:03.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBThuBmfMfFJhAAOe1CTS5SFsOwQ5TGQtYimg=="}' 2024-09-10T17:48:03.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69658764-ae7c-4ad3-9db0-3aaa3c2162dc -i td/ceph-helpers/0/new.json 2024-09-10T17:48:03.934 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:48:03.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:48:03.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBThuBmfMfFJhAAOe1CTS5SFsOwQ5TGQtYimg== --osd-uuid 69658764-ae7c-4ad3-9db0-3aaa3c2162dc 2024-09-10T17:48:03.974 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:03.973+0000 7f0d01044640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:04.003 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:04.002+0000 7f0d01044640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:04.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:04.003+0000 7f0d01044640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:04.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:04.003+0000 7f0d01044640 -1 bdev(0x55b637aa3c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:48:04.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:04.003+0000 7f0d01044640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:48:09.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:48:09.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:48:09.735 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:48:09.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:48:09.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:48:10.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:48:10.318 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:48:10.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:48:10.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:48:10.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:48:10.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:48:10.345 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:10.344+0000 7fb9d6640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:10.365 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:10.363+0000 7fb9d6640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:10.366 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:10.365+0000 7fb9d6640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:10.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:48:10.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:48:10.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:48:10.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:48:10.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:48:10.735 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:48:10.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:10.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:48:10.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:10.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:11.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:11.450 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:11.449+0000 7fb9d6640640 -1 Falling back to public interface 2024-09-10T17:48:12.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:12.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:12.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:48:12.165 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:48:12.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:12.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:12.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:13.267 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:13.266+0000 7fb9d6640640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:48:13.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:13.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:13.590 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:48:13.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:48:13.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:13.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:13.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:14.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:14.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:14.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:48:14.986 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:48:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:15.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:16.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:16.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:16.421 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:48:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:48:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:16.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:17.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:17.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:17.820 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:48:17.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:48:17.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:17.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:18.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:19.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:19.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:19.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:48:19.305 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:48:19.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:19.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:19.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:20.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:20.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:20.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:48:20.705 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:48:20.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:20.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:21.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:22.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:22.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:22.141 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:48:22.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:48:22.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:22.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:22.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:23.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:23.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:23.564 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:48:23.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:48:23.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:23.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:24.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:25.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:25.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:25.103 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:48:25.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:48:25.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:25.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:25.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:26.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:26.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:26.823 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:48:26.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:48:26.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:26.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:27.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:28.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:28.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:28.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:48:28.335 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:48:28.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:28.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:28.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:29.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:29.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:29.973 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:48:29.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:48:29.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:29.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:31.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:32.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:32.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:32.042 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:48:32.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:48:32.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:32.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:32.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:33.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:33.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:33.473 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:48:33.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:48:33.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:33.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:33.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:34.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:34.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:34.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:48:34.873 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:48:34.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:34.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:48:35.376 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2325732525,v1:127.0.0.1:6803/2325732525] [v2:127.0.0.1:6804/2325732525,v1:127.0.0.1:6805/2325732525] exists,up 69658764-ae7c-4ad3-9db0-3aaa3c2162dc 2024-09-10T17:48:35.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:48:35.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1028: test_get_osds: run_osd td/ceph-helpers 1 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-10T17:48:35.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:48:35.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=85637182-4263-4dfa-acb7-d3e433e32535 2024-09-10T17:48:35.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 85637182-4263-4dfa-acb7-d3e433e32535' 2024-09-10T17:48:35.384 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 85637182-4263-4dfa-acb7-d3e433e32535 2024-09-10T17:48:35.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:48:35.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBzhuBmA3e5FxAAQwArq46WWM23Sl0+bYcdyQ== 2024-09-10T17:48:35.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBzhuBmA3e5FxAAQwArq46WWM23Sl0+bYcdyQ=="}' 2024-09-10T17:48:35.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 85637182-4263-4dfa-acb7-d3e433e32535 -i td/ceph-helpers/1/new.json 2024-09-10T17:48:35.873 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:48:35.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2024-09-10T17:48:35.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzhuBmA3e5FxAAQwArq46WWM23Sl0+bYcdyQ== --osd-uuid 85637182-4263-4dfa-acb7-d3e433e32535 2024-09-10T17:48:35.915 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:35.914+0000 7f866f919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:35.923 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:35.922+0000 7f866f919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:35.924 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:35.923+0000 7f866f919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:35.924 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:35.923+0000 7f866f919640 -1 bdev(0x565351d23c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:48:35.924 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:35.923+0000 7f866f919640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-10T17:48:42.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-10T17:48:42.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:48:42.364 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-10T17:48:42.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:48:42.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:48:42.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:48:42.949 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-10T17:48:42.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:48:42.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:48:42.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:48:42.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:48:42.976 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:42.975+0000 7fd0d3302640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:42.978 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:42.977+0000 7fd0d3302640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:42.979 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:42.978+0000 7fd0d3302640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:43.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:48:43.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:48:43.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:48:43.376 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:48:43.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:48:43.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:48:43.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:43.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:48:43.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:43.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:43.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:44.321 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:44.320+0000 7fd0d3302640 -1 Falling back to public interface 2024-09-10T17:48:44.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:44.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:44.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:48:44.777 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:48:44.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:44.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:45.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:46.191 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:48:46.190+0000 7fd0d3302640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:48:46.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:46.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:46.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:48:46.214 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:48:46.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:46.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:46.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:47.828 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:48:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:48:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:48.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:49.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:49.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:49.248 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:48:49.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:48:49.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:49.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:49.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:50.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:50.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:50.670 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:48:50.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:48:50.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:50.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:51.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:52.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:52.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:52.076 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:48:52.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:48:52.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:52.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:52.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:53.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:53.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:53.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:48:53.581 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:48:53.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:53.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:54.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:55.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:55.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:55.081 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:48:55.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:48:55.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:55.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:55.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:56.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:56.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:56.484 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:48:56.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:48:56.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:56.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:56.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:57.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:57.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:57.967 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:48:57.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:48:57.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:57.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:48:58.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:59.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:59.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:59.852 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:48:59.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:48:59.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:59.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:00.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:01.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:01.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:01.571 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:49:01.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:49:01.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:01.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:02.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:03.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:03.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:03.274 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:49:03.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:49:03.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:03.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:03.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:04.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:04.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:04.909 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:49:04.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:49:04.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:04.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:05.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:06.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:06.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:06.670 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:49:06.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:49:06.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:06.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:07.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:08.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:08.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:08.548 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:49:08.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:49:08.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:08.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:09.038 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2736864920,v1:127.0.0.1:6811/2736864920] [v2:127.0.0.1:6812/2736864920,v1:127.0.0.1:6813/2736864920] exists,up 85637182-4263-4dfa-acb7-d3e433e32535 2024-09-10T17:49:09.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:49:09.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:49:09.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:49:09.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: test_get_osds: create_rbd_pool 2024-09-10T17:49:09.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:49:09.450 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:49:09.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:49:09.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:49:10.261 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:49:10.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:49:11.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:49:12.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: test_get_osds: wait_for_clean 2024-09-10T17:49:12.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:49:12.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:49:12.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:49:12.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:49:12.670 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:49:12.670 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:49:12.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:49:12.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:49:12.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:49:12.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:49:12.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:49:12.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:49:12.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:49:12.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:49:12.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:49:13.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:49:13.187 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2024-09-10T17:49:13.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:49:13.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:13.187 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:49:13.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836494 2024-09-10T17:49:13.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836494 2024-09-10T17:49:13.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494' 2024-09-10T17:49:13.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:13.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:49:13.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-10T17:49:13.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-10T17:49:13.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494 1-42949672965' 2024-09-10T17:49:13.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:13.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836494 2024-09-10T17:49:13.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:13.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:49:13.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836494 2024-09-10T17:49:13.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:13.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836494 2024-09-10T17:49:13.419 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836494 2024-09-10T17:49:13.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-10T17:49:13.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:49:13.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-10T17:49:13.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:13.847 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-10T17:49:13.847 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:13.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:49:13.848 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-10T17:49:13.848 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:13.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-10T17:49:13.849 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672965 2024-09-10T17:49:13.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-10T17:49:13.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:49:14.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-10T17:49:14.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:49:14.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:49:14.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:49:14.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:49:14.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:49:14.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:49:14.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:49:14.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:49:14.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:49:14.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:49:14.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:49:15.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:49:15.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:49:15.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:49:15.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:49:15.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:49:15.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:49:15.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:49:15.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: create_rbd_pool 2024-09-10T17:49:15.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:49:16.255 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:49:16.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:49:16.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:49:16.877 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:49:16.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:49:17.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:49:19.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: get_osds rbd GROUP 2024-09-10T17:49:19.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-09-10T17:49:19.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=rbd 2024-09-10T17:49:19.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=GROUP 2024-09-10T17:49:19.479 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map rbd GROUP 2024-09-10T17:49:19.479 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:49:19.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:0' 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: teardown td/ceph-helpers 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:49:19.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:49:20.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:49:20.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:49:20.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:49:20.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:49:20.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:49:20.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:49:20.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:49:20.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:49:20.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:49:20.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:49:20.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:49:20.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:49:20.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:49:20.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:49:20.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:49:20.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:20.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:49:20.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:49:20.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:49:20.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:49:20.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:49:20.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_pg td/ceph-helpers 2024-09-10T17:49:20.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1091: test_get_pg: local dir=td/ceph-helpers 2024-09-10T17:49:20.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: test_get_pg: setup td/ceph-helpers 2024-09-10T17:49:20.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:49:20.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:49:20.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:49:20.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:49:20.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:49:20.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:49:20.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:49:20.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:49:20.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:49:20.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:49:20.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:49:20.051 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:49:20.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:49:20.052 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:49:20.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:49:20.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:49:20.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:49:20.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:49:20.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:49:20.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:49:20.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:49:20.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:49:20.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:49:20.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:49:20.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:49:20.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:20.061 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:49:20.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:49:20.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:49:20.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:49:20.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:49:20.061 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:49:20.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:20.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:49:20.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:49:20.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:49:20.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:49:20.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:49:20.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:49:20.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:49:20.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:49:20.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:49:20.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:49:20.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:49:20.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:49:20.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:49:20.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:49:20.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:49:20.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:49:20.517 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:49:20.517 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:20.517 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:49:20.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:49:20.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:49:20.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:49:20.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:49:20.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:49:20.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:49:20.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:49:20.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:49:20.711 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:49:20.711 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:49:20.711 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:49:20.712 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:49:20.712 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:20.712 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:49:20.712 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:49:20.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:49:20.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:49:20.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:49:20.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:20.795 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:49:20.796 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:49:20.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:49:20.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:49:20.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1095: test_get_pg: run_mgr td/ceph-helpers x 2024-09-10T17:49:20.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:49:20.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:49:20.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:49:20.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:49:20.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:49:20.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:49:21.721 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:49:21.721 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:49:21.721 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:49:21.721 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:49:21.721 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:21.721 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:49:21.721 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:49:21.722 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:49:21.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:49:21.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: test_get_pg: run_osd td/ceph-helpers 0 2024-09-10T17:49:21.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:49:21.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:49:21.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:49:21.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:49:21.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:49:21.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:21.753 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:49:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:49:21.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:49:21.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:49:21.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:49:21.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a35d1ddc-3e67-4a8f-9a35-47bc8dc91730 2024-09-10T17:49:21.755 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 a35d1ddc-3e67-4a8f-9a35-47bc8dc91730 2024-09-10T17:49:21.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a35d1ddc-3e67-4a8f-9a35-47bc8dc91730' 2024-09-10T17:49:21.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:49:21.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQChhuBmsuj/LRAAIdej0hJtQ2uWvP63OLt5MA== 2024-09-10T17:49:21.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQChhuBmsuj/LRAAIdej0hJtQ2uWvP63OLt5MA=="}' 2024-09-10T17:49:21.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a35d1ddc-3e67-4a8f-9a35-47bc8dc91730 -i td/ceph-helpers/0/new.json 2024-09-10T17:49:22.049 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:49:22.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:49:22.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChhuBmsuj/LRAAIdej0hJtQ2uWvP63OLt5MA== --osd-uuid a35d1ddc-3e67-4a8f-9a35-47bc8dc91730 2024-09-10T17:49:22.089 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:22.088+0000 7ff6ee525640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:22.118 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:22.117+0000 7ff6ee525640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:22.119 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:22.118+0000 7ff6ee525640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:22.119 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:22.118+0000 7ff6ee525640 -1 bdev(0x556a94ce3c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:49:22.119 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:22.118+0000 7ff6ee525640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:49:28.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:49:28.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:49:28.045 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:49:28.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:49:28.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:49:28.582 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:49:28.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:49:28.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:49:28.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:49:28.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:49:28.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:49:28.609 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:28.608+0000 7f0fa3725640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:28.620 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:28.619+0000 7f0fa3725640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:28.621 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:28.620+0000 7f0fa3725640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:29.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:49:29.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:49:29.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:49:29.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:49:29.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:49:29.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:29.034 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:49:29.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:49:29.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:29.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:29.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:29.969 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:29.969+0000 7f0fa3725640 -1 Falling back to public interface 2024-09-10T17:49:30.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:30.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:30.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:49:30.461 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:49:30.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:30.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:30.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:31.582 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:49:31.580+0000 7f0fa3725640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:49:31.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:31.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:31.896 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:49:31.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:49:31.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:31.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:32.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:33.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:33.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:33.324 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:49:33.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:49:33.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:33.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:33.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:34.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:34.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:34.749 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:49:34.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:49:34.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:34.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:35.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:36.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:36.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:36.152 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:49:36.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:49:36.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:36.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:36.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:37.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:37.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:37.648 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:49:37.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:49:37.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:37.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:38.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:39.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:39.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:39.065 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:49:39.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:49:39.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:39.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:39.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:40.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:40.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:40.470 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:49:40.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:49:40.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:40.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:40.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:41.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:41.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:41.881 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:49:41.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:49:41.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:41.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:42.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:43.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:43.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:43.389 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:49:43.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:49:43.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:43.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:43.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:44.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:44.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:44.928 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:49:44.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:49:44.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:44.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:45.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:46.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:46.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:46.513 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:49:46.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:49:46.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:46.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:47.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:48.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:48.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:48.149 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:49:48.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:49:48.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:48.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:48.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:49.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:49.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:49.993 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:49:49.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:49:49.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:49.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:50.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:51.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:51.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:51.788 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:49:51.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:49:51.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:51.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:52.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:53.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:53.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:53.229 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:49:53.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:49:53.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:53.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:54.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:55.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:55.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:55.018 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:49:55.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:49:55.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:55.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:55.440 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1514525662,v1:127.0.0.1:6803/1514525662] [v2:127.0.0.1:6804/1514525662,v1:127.0.0.1:6805/1514525662] exists,up a35d1ddc-3e67-4a8f-9a35-47bc8dc91730 2024-09-10T17:49:55.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:49:55.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:49:55.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:49:55.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1097: test_get_pg: create_rbd_pool 2024-09-10T17:49:55.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:49:55.827 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:49:55.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:49:55.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:49:56.548 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:49:56.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:49:57.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:49:58.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: test_get_pg: wait_for_clean 2024-09-10T17:49:58.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:49:58.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:49:58.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:49:58.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:49:58.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:49:58.552 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:49:58.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:49:58.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:49:58.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:49:58.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:49:58.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:49:58.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:49:58.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:49:58.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:49:58.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:49:59.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:49:59.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:49:59.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:59.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:49:59.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836486 2024-09-10T17:49:59.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836486 2024-09-10T17:49:59.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836486' 2024-09-10T17:49:59.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:59.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836486 2024-09-10T17:49:59.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:59.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:49:59.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836486 2024-09-10T17:49:59.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:59.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836486 2024-09-10T17:49:59.193 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-10T17:49:59.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-10T17:49:59.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:49:59.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-10T17:49:59.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:50:00.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:50:00.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:50:01.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-10T17:50:01.038 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:50:01.038 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:50:01.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:50:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:50:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:50:01.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:50:01.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:50:01.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:50:01.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:50:01.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:50:01.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:50:02.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:50:02.028 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:50:02.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:50:02.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:50:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:50:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:50:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:50:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: get_pg rbd GROUP 2024-09-10T17:50:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-10T17:50:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=GROUP 2024-09-10T17:50:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-09-10T17:50:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd GROUP 2024-09-10T17:50:02.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T17:50:02.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: teardown td/ceph-helpers 2024-09-10T17:50:02.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:50:02.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:50:02.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:50:02.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:50:02.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:50:02.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:50:02.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:50:02.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:50:03.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:50:03.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:50:03.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:50:03.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:50:03.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:50:03.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:50:03.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:50:03.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:50:03.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:50:03.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:50:03.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:50:03.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:50:03.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:50:03.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:50:03.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:50:03.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:03.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:03.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:50:03.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:50:03.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:50:03.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:50:03.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_primary td/ceph-helpers 2024-09-10T17:50:03.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: test_get_primary: local dir=td/ceph-helpers 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: test_get_primary: setup td/ceph-helpers 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:50:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:50:03.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:50:03.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:50:03.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:50:03.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:50:03.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:50:03.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:50:03.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:50:03.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:50:03.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:50:03.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:50:03.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:50:03.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:50:03.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:50:03.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:50:03.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:50:03.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:03.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:03.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:50:03.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:50:03.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:50:03.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:50:03.088 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:50:03.088 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:03.088 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:03.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:50:03.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:50:03.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:50:03.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:50:03.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:50:03.590 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:50:03.590 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:03.590 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:03.591 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:50:03.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:50:03.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:50:03.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:50:03.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:50:03.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:50:03.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:50:03.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:50:03.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:50:03.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:50:03.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:50:03.784 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:50:03.785 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:03.785 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:03.785 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:50:03.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:50:03.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:50:03.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:50:03.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:50:03.864 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:50:03.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: test_get_primary: local osd=0 2024-09-10T17:50:03.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: test_get_primary: run_mgr td/ceph-helpers x 2024-09-10T17:50:03.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:50:03.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:50:03.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:50:03.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:50:03.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:50:03.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:50:04.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:50:04.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:50:04.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:50:04.785 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:50:04.785 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:04.786 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:04.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:50:04.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:50:04.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:50:04.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: test_get_primary: run_osd td/ceph-helpers 0 2024-09-10T17:50:04.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:50:04.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:50:04.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:50:04.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:50:04.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:50:04.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:50:04.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:50:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:50:04.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:50:04.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:50:04.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:50:04.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c61f17c9-0b50-4096-984f-c7b4e6ec35c4 2024-09-10T17:50:04.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c61f17c9-0b50-4096-984f-c7b4e6ec35c4' 2024-09-10T17:50:04.820 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 c61f17c9-0b50-4096-984f-c7b4e6ec35c4 2024-09-10T17:50:04.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:50:04.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMhuBmMRPMMRAAcbQ6t7r+5+zy5nkdl1wy+Q== 2024-09-10T17:50:04.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMhuBmMRPMMRAAcbQ6t7r+5+zy5nkdl1wy+Q=="}' 2024-09-10T17:50:04.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c61f17c9-0b50-4096-984f-c7b4e6ec35c4 -i td/ceph-helpers/0/new.json 2024-09-10T17:50:05.113 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:50:05.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:50:05.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMhuBmMRPMMRAAcbQ6t7r+5+zy5nkdl1wy+Q== --osd-uuid c61f17c9-0b50-4096-984f-c7b4e6ec35c4 2024-09-10T17:50:05.153 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:05.152+0000 7fa60c4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:05.182 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:05.181+0000 7fa60c4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:05.183 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:05.182+0000 7fa60c4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:05.183 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:05.183+0000 7fa60c4f1640 -1 bdev(0x564ee92cbc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:50:05.184 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:05.183+0000 7fa60c4f1640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:50:10.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:50:10.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:50:10.705 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:50:10.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:50:10.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:50:11.251 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:50:11.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:50:11.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:50:11.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:50:11.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:50:11.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:50:11.279 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:11.278+0000 7ff9e05fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:11.281 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:11.280+0000 7ff9e05fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:11.282 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:11.281+0000 7ff9e05fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:11.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:50:11.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:50:11.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:50:11.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:50:11.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:50:11.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:11.683 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:50:11.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:50:11.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:11.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:12.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:12.634 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:12.633+0000 7ff9e05fd640 -1 Falling back to public interface 2024-09-10T17:50:13.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:13.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:13.064 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:50:13.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:50:13.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:13.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:13.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:14.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:14.444+0000 7ff9e05fd640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:50:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:14.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:14.502 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:50:14.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:50:14.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:14.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:15.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:16.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:16.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:16.158 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:50:16.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:50:16.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:16.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:16.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:17.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:17.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:17.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:50:17.603 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:50:17.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:17.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:18.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:19.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:19.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:19.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:50:19.010 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:50:19.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:19.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:19.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:20.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:20.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:20.429 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:50:20.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:50:20.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:20.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:20.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:21.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:21.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:21.853 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:50:21.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:50:21.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:21.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:22.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:23.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:23.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:23.303 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:50:23.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:50:23.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:23.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:23.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:24.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:24.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:24.706 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:50:24.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:50:24.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:24.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:25.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:26.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:26.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:26.101 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:50:26.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:50:26.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:26.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:26.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:27.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:27.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:27.868 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:50:27.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:50:27.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:27.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:28.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:29.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:29.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:29.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:50:29.446 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:50:29.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:29.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:30.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:31.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:31.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:31.206 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:50:31.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:50:31.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:31.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:31.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:32.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:32.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:32.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:50:32.812 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:50:32.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:32.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:33.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:34.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:34.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:34.382 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:50:34.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:50:34.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:34.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:35.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:36.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:36.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:36.082 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:50:36.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:50:36.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:36.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:36.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:37.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:37.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:37.960 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:50:37.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:50:37.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:37.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:38.385 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1336928254,v1:127.0.0.1:6803/1336928254] [v2:127.0.0.1:6804/1336928254,v1:127.0.0.1:6805/1336928254] exists,up c61f17c9-0b50-4096-984f-c7b4e6ec35c4 2024-09-10T17:50:38.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:50:38.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:50:38.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:50:38.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: test_get_primary: create_rbd_pool 2024-09-10T17:50:38.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:50:38.823 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:50:38.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:50:38.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:50:39.358 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:50:39.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:50:40.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:50:41.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: test_get_primary: wait_for_clean 2024-09-10T17:50:41.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:50:41.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:50:41.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:50:41.381 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:50:41.382 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:41.382 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:41.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:41.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:41.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:41.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:50:41.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:50:41.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:50:41.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:50:41.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:50:41.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:50:41.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:50:41.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:50:41.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:50:41.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:50:42.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836486 2024-09-10T17:50:42.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836486 2024-09-10T17:50:42.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836486' 2024-09-10T17:50:42.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:50:42.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836486 2024-09-10T17:50:42.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:50:42.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:50:42.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836486 2024-09-10T17:50:42.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:50:42.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836486 2024-09-10T17:50:42.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-10T17:50:42.025 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-10T17:50:42.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:50:42.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-10T17:50:42.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:50:43.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:50:43.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:50:43.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-10T17:50:43.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:50:43.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:50:43.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:50:44.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:50:44.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:50:44.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:50:44.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:50:44.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:50:44.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:50:44.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:50:44.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:50:44.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:50:44.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:50:44.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:50:44.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:50:45.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:50:45.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:50:45.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:50:45.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: get_primary rbd GROUP 2024-09-10T17:50:45.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-10T17:50:45.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2024-09-10T17:50:45.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2024-09-10T17:50:45.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T17:50:45.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: test 0 = 0 2024-09-10T17:50:45.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: test_get_primary: teardown td/ceph-helpers 2024-09-10T17:50:45.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:50:45.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:50:45.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:50:45.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:50:45.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:50:45.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:50:45.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:50:45.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:50:45.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:50:45.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:50:45.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:50:45.938 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:50:45.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:50:45.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:50:45.940 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:50:45.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:50:45.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:50:45.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:50:45.941 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:50:45.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:50:45.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:50:45.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:50:45.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:50:45.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:45.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:45.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:50:45.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:50:45.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:50:45.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:50:45.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_timeout_delays td/ceph-helpers 2024-09-10T17:50:45.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: test_get_timeout_delays: get_timeout_delays 1 2024-09-10T17:50:45.951 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:45.951 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:45.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:45.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:45.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:45.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: test_get_timeout_delays: test '1 ' = '1 ' 2024-09-10T17:50:45.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: test_get_timeout_delays: get_timeout_delays 5 2024-09-10T17:50:45.962 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:45.962 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:45.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:45.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:45.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:45.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-09-10T17:50:45.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: test_get_timeout_delays: get_timeout_delays 6 2024-09-10T17:50:45.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:45.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:45.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:45.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:45.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:45.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-09-10T17:50:46.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: test_get_timeout_delays: get_timeout_delays 7 2024-09-10T17:50:46.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-09-10T17:50:46.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: test_get_timeout_delays: get_timeout_delays 8 2024-09-10T17:50:46.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-09-10T17:50:46.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: test_get_timeout_delays: get_timeout_delays 1 .1 2024-09-10T17:50:46.050 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.050 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-09-10T17:50:46.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-09-10T17:50:46.076 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.076 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-09-10T17:50:46.106 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1614: test_get_timeout_delays: get_timeout_delays 5 .1 2024-09-10T17:50:46.107 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.107 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1614: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-09-10T17:50:46.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: test_get_timeout_delays: get_timeout_delays 6 .1 2024-09-10T17:50:46.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-09-10T17:50:46.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-09-10T17:50:46.186 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.186 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-09-10T17:50:46.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 20 .1 2024-09-10T17:50:46.224 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.224 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-09-10T17:50:46.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-09-10T17:50:46.277 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.277 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-09-10T17:50:46.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-09-10T17:50:46.356 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:50:46.356 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:50:46.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:50:46.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:50:46.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:50:46.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-09-10T17:50:46.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:50:46.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_is_clean td/ceph-helpers 2024-09-10T17:50:46.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: test_is_clean: local dir=td/ceph-helpers 2024-09-10T17:50:46.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: test_is_clean: setup td/ceph-helpers 2024-09-10T17:50:46.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:50:46.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:50:46.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:50:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:50:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:50:46.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:50:46.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:50:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:50:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:50:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:50:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:50:46.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:50:46.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:50:46.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:50:46.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:50:46.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:50:46.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:50:46.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:50:46.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:50:46.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:50:46.598 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:50:46.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:50:46.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:50:46.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:50:46.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:50:46.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:46.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:46.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:50:46.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:50:46.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:50:46.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:50:46.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:50:46.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:46.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:46.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:50:46.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:50:46.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:50:46.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:50:46.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:50:46.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:50:46.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:50:46.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:50:46.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:50:46.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:50:46.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:50:46.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:50:47.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:50:47.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:50:47.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:50:47.077 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:50:47.077 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:47.077 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:47.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:50:47.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:50:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:50:47.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:50:47.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:50:47.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:50:47.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:47.343 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:50:47.344 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:50:47.344 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:50:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: test_is_clean: run_mgr td/ceph-helpers x 2024-09-10T17:50:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:50:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:50:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:50:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:50:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:50:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:50:48.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:50:48.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:50:48.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:50:48.266 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:50:48.266 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:48.266 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:48.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:50:48.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:50:48.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:50:48.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: test_is_clean: run_osd td/ceph-helpers 0 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:50:48.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:50:48.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:50:48.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:50:48.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:50:48.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:50:48.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:50:48.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:50:48.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:50:48.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:50:48.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:50:48.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:50:48.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0c26b407-8361-42e2-815d-4d4d6a517781 2024-09-10T17:50:48.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0c26b407-8361-42e2-815d-4d4d6a517781' 2024-09-10T17:50:48.300 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 0c26b407-8361-42e2-815d-4d4d6a517781 2024-09-10T17:50:48.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:50:48.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD4huBmvpDZEhAAeEMcZSl9OJZ4sxHFrP3NOQ== 2024-09-10T17:50:48.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD4huBmvpDZEhAAeEMcZSl9OJZ4sxHFrP3NOQ=="}' 2024-09-10T17:50:48.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0c26b407-8361-42e2-815d-4d4d6a517781 -i td/ceph-helpers/0/new.json 2024-09-10T17:50:48.591 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:50:48.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:50:48.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4huBmvpDZEhAAeEMcZSl9OJZ4sxHFrP3NOQ== --osd-uuid 0c26b407-8361-42e2-815d-4d4d6a517781 2024-09-10T17:50:48.632 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:48.631+0000 7fadfc42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:48.638 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:48.637+0000 7fadfc42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:48.639 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:48.638+0000 7fadfc42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:48.639 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:48.638+0000 7fadfc42c640 -1 bdev(0x55ca3c477c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:50:48.640 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:48.639+0000 7fadfc42c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:50:54.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:50:54.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:50:54.372 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:50:54.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:50:54.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:50:54.919 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:50:54.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:50:54.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:50:54.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:50:54.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:50:54.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:50:54.946 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:54.945+0000 7f4841a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:54.968 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:54.967+0000 7f4841a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:54.969 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:54.968+0000 7f4841a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:55.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:50:55.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:50:55.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:50:55.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:50:55.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:50:55.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:55.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:50:55.350 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:50:55.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:55.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:55.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:55.819+0000 7f4841a1d640 -1 Falling back to public interface 2024-09-10T17:50:56.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:56.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:56.767 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:50:56.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:50:56.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:56.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:57.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:57.561 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:50:57.560+0000 7f4841a1d640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:50:58.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:58.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:58.164 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:50:58.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:50:58.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:58.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:58.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:59.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:59.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:59.662 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:50:59.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:50:59.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:59.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:00.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:01.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:01.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:01.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:51:01.092 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:51:01.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:01.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:01.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:02.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:02.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:02.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:51:02.623 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:51:02.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:02.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:03.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:04.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:04.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:04.054 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:51:04.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:51:04.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:04.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:04.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:05.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:05.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:05.489 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:51:05.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:51:05.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:05.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:05.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:06.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:06.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:06.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:51:06.887 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:51:06.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:06.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:07.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:08.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:08.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:08.443 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:51:08.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:51:08.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:08.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:08.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:09.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:09.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:09.865 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:51:09.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:51:09.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:09.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:10.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:11.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:11.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:11.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:51:11.278 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:51:11.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:11.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:12.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:13.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:13.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:13.097 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:51:13.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:51:13.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:13.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:13.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:14.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:14.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:14.891 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:51:14.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:51:14.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:14.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:15.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:16.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:16.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:16.830 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:51:16.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:51:16.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:16.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:17.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:18.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:18.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:18.876 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:51:18.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:51:18.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:18.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:19.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:20.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:20.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:20.393 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:51:20.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:51:20.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:20.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:20.888 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1410847537,v1:127.0.0.1:6803/1410847537] [v2:127.0.0.1:6804/1410847537,v1:127.0.0.1:6805/1410847537] exists,up 0c26b407-8361-42e2-815d-4d4d6a517781 2024-09-10T17:51:20.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:51:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:51:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:51:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1558: test_is_clean: create_rbd_pool 2024-09-10T17:51:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:51:21.296 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:51:21.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:51:21.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:51:21.797 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:51:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:51:22.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:51:23.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: test_is_clean: wait_for_clean 2024-09-10T17:51:23.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:51:23.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:51:23.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:51:23.853 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:51:23.853 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:51:23.853 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:51:23.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:51:23.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:51:23.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:51:23.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:51:23.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:51:23.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:51:23.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:51:23.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:51:23.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:51:24.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:51:24.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:51:24.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:51:24.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:51:24.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836484 2024-09-10T17:51:24.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836484 2024-09-10T17:51:24.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836484' 2024-09-10T17:51:24.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:51:24.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836484 2024-09-10T17:51:24.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:51:24.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:51:24.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836484 2024-09-10T17:51:24.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:51:24.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836484 2024-09-10T17:51:24.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-10T17:51:24.460 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836484 2024-09-10T17:51:24.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:51:24.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-10T17:51:24.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:51:25.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:51:25.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:51:26.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-10T17:51:26.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:51:27.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T17:51:27.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:51:27.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-10T17:51:27.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:51:27.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:51:27.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:51:28.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:51:28.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:51:28.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:51:28.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:51:28.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:51:28.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:51:28.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:51:28.331 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:51:28.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:51:28.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:51:28.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:51:28.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:51:29.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:51:29.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:51:29.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:51:29.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1560: test_is_clean: is_clean 2024-09-10T17:51:29.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: get_num_pgs 2024-09-10T17:51:29.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:51:29.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:51:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: num_pgs=4 2024-09-10T17:51:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1547: is_clean: test 4 '!=' 0 2024-09-10T17:51:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: get_num_active_clean 2024-09-10T17:51:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:51:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:51:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:51:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:51:29.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:51:30.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: test 4 = 4 2024-09-10T17:51:30.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: teardown td/ceph-helpers 2024-09-10T17:51:30.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:51:30.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:51:30.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:51:30.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:51:30.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:51:30.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:51:30.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:51:30.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:51:30.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:51:30.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:51:30.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:51:30.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:51:30.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:51:30.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:51:30.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:51:30.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:51:30.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:51:30.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:51:30.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:51:30.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:51:30.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:51:30.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:51:30.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:51:30.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:30.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:51:30.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:51:30.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:51:30.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:51:30.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:51:30.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_kill_daemon td/ceph-helpers 2024-09-10T17:51:30.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: test_kill_daemon: local dir=td/ceph-helpers 2024-09-10T17:51:30.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:267: test_kill_daemon: setup td/ceph-helpers 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:51:30.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:51:30.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:51:30.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:51:30.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:51:30.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:51:30.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:51:30.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:51:30.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:51:30.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:51:30.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:51:30.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:51:30.315 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:51:30.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:51:30.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:51:30.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:51:30.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:51:30.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:30.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:51:30.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:51:30.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:51:30.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:51:30.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:51:30.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:51:30.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:30.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:51:30.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:51:30.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:51:30.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:51:30.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:51:30.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:51:30.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:51:30.768 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:51:30.768 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:30.768 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:51:30.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:51:30.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:51:30.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:30.945 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:51:30.946 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:51:30.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:51:30.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:51:31.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:51:31.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:51:31.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:51:31.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:51:31.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:51:31.032 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:51:31.032 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:51:31.032 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:51:31.032 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:51:31.033 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:31.033 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:51:31.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:51:31.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:51:31.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:51:31.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: test_kill_daemon: run_mgr td/ceph-helpers x 2024-09-10T17:51:31.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:51:31.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:51:31.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:51:31.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:51:31.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:51:31.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:51:31.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:51:31.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:51:31.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:51:31.966 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:51:31.966 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:31.966 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:51:31.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:51:31.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:51:31.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:51:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:270: test_kill_daemon: run_osd td/ceph-helpers 0 2024-09-10T17:51:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:51:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:51:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:51:31.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:51:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:51:32.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:51:32.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:51:32.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:51:32.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:51:32.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:51:32.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:51:32.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:51:32.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c8b3ac36-91c5-4a89-b019-ffbd76c81cc3 2024-09-10T17:51:32.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c8b3ac36-91c5-4a89-b019-ffbd76c81cc3' 2024-09-10T17:51:32.001 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 c8b3ac36-91c5-4a89-b019-ffbd76c81cc3 2024-09-10T17:51:32.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:51:32.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAkh+Bm8RD9ABAAsD98j14xUFjfOim5I2SF2A== 2024-09-10T17:51:32.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAkh+Bm8RD9ABAAsD98j14xUFjfOim5I2SF2A=="}' 2024-09-10T17:51:32.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c8b3ac36-91c5-4a89-b019-ffbd76c81cc3 -i td/ceph-helpers/0/new.json 2024-09-10T17:51:32.285 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:51:32.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:51:32.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkh+Bm8RD9ABAAsD98j14xUFjfOim5I2SF2A== --osd-uuid c8b3ac36-91c5-4a89-b019-ffbd76c81cc3 2024-09-10T17:51:32.325 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:32.324+0000 7f76ede16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:32.355 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:32.353+0000 7f76ede16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:32.355 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:32.354+0000 7f76ede16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:32.355 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:32.354+0000 7f76ede16640 -1 bdev(0x55b8a2a4bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:51:32.356 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:32.354+0000 7f76ede16640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:51:38.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:51:38.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:51:38.075 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:51:38.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:51:38.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:51:38.655 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:51:38.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:51:38.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:51:38.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:51:38.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:51:38.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:51:38.689 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:38.683+0000 7f9437a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:38.700 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:38.699+0000 7f9437a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:38.701 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:38.700+0000 7f9437a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:39.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:51:39.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:51:39.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:51:39.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:51:39.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:51:39.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:39.114 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:51:39.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:51:39.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:39.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:39.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:39.557 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:39.556+0000 7f9437a15640 -1 Falling back to public interface 2024-09-10T17:51:40.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:40.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:40.532 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:51:40.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:51:40.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:40.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:40.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:41.652 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:51:41.651+0000 7f9437a15640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:51:41.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:41.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:41.922 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:51:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:51:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:42.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:43.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:43.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:43.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:51:43.394 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:51:43.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:43.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:44.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:44.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:44.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:51:44.791 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:51:44.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:44.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:45.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:46.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:46.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:46.200 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:51:46.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:51:46.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:46.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:47.598 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:48.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:49.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:49.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:49.041 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:51:49.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:51:49.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:49.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:49.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:50.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:50.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:50.549 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:51:50.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:51:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:50.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:51.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:51.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:51.945 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:51:51.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:51:51.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:51.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:52.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:53.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:53.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:53.406 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:51:53.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:51:53.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:53.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:54.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:55.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:55.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:55.167 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:51:55.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:51:55.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:55.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:55.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:56.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:56.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:56.688 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:51:56.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:51:56.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:56.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:57.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:58.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:58.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:58.420 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:51:58.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:51:58.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:58.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:51:59.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:00.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:00.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:00.344 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:52:00.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:52:00.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:00.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:00.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:01.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:01.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:01.831 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:52:01.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:52:01.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:01.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:02.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:03.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:03.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:03.373 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:52:03.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:52:03.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:03.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:03.983 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2492039284,v1:127.0.0.1:6803/2492039284] [v2:127.0.0.1:6804/2492039284,v1:127.0.0.1:6805/2492039284] exists,up c8b3ac36-91c5-4a89-b019-ffbd76c81cc3 2024-09-10T17:52:03.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:52:03.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:52:03.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:52:03.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:272: test_kill_daemon: name_prefix=osd 2024-09-10T17:52:03.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: find td/ceph-helpers 2024-09-10T17:52:03.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-09-10T17:52:03.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-10T17:52:03.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:280: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-09-10T17:52:03.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-10T17:52:03.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=122767 2024-09-10T17:52:03.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=0 2024-09-10T17:52:03.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local delays=1 2024-09-10T17:52:03.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2024-09-10T17:52:03.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:03.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 122767 2024-09-10T17:52:03.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-10T17:52:03.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-10T17:52:03.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 1 2024-09-10T17:52:04.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 122767 2024-09-10T17:52:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-10T17:52:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-10T17:52:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0 2024-09-10T17:52:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 1 2024-09-10T17:52:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:284: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-09-10T17:52:04.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-10T17:52:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=122767 2024-09-10T17:52:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2024-09-10T17:52:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-10T17:52:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2024-09-10T17:52:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 122767 2024-09-10T17:52:04.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-10T17:52:04.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-10T17:52:04.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2024-09-10T17:52:05.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:05.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 122767 2024-09-10T17:52:05.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-10T17:52:05.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-10T17:52:05.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2024-09-10T17:52:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 122767 2024-09-10T17:52:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2024-09-10T17:52:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2024-09-10T17:52:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2024-09-10T17:52:05.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:287: test_kill_daemon: name_prefix=mgr 2024-09-10T17:52:05.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: find td/ceph-helpers 2024-09-10T17:52:05.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-09-10T17:52:05.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-10T17:52:05.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:292: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-09-10T17:52:05.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=121817 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 121817 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-10T17:52:05.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2024-09-10T17:52:05.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:05.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 121817 2024-09-10T17:52:05.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2024-09-10T17:52:05.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2024-09-10T17:52:05.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2024-09-10T17:52:05.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:295: test_kill_daemon: name_prefix=mon 2024-09-10T17:52:05.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: find td/ceph-helpers 2024-09-10T17:52:05.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-09-10T17:52:05.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-10T17:52:05.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:300: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-09-10T17:52:05.400 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-09-10T17:52:05.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=121732 2024-09-10T17:52:05.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2024-09-10T17:52:05.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-10T17:52:05.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2024-09-10T17:52:05.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:05.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 121732 2024-09-10T17:52:05.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-10T17:52:05.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-10T17:52:05.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2024-09-10T17:52:05.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-10T17:52:05.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 121732 2024-09-10T17:52:05.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2024-09-10T17:52:05.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2024-09-10T17:52:05.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2024-09-10T17:52:05.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: timeout 5 ceph status 2024-09-10T17:52:10.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: teardown td/ceph-helpers 2024-09-10T17:52:10.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:52:10.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:52:10.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:52:10.507 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:10.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:10.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:10.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:10.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:10.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:10.513 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:52:10.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:52:10.514 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:52:10.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:52:10.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:52:10.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:52:10.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:10.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:52:10.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:52:10.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:10.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:52:10.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:52:10.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:52:10.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:52:10.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:10.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:10.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_kill_daemons td/ceph-helpers 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:357: test_kill_daemons: local dir=td/ceph-helpers 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:358: test_kill_daemons: setup td/ceph-helpers 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:52:10.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:52:10.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:52:10.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:10.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:10.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:10.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:10.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:10.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:10.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:52:10.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:52:10.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:52:10.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:52:10.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:52:10.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:52:10.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:10.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:52:10.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:52:10.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:10.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:52:10.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:52:10.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:52:10.537 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:52:10.537 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:10.537 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:10.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:52:10.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:52:10.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:52:10.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:52:10.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:52:10.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:10.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:10.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:52:10.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:52:10.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:52:10.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:52:10.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:52:10.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:359: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:52:10.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:52:10.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:52:10.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:52:10.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:52:10.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:52:10.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:52:11.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:52:11.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:11.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:11.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:11.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:11.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:11.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:52:11.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:52:11.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:52:11.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:52:11.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:52:11.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:52:11.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:52:11.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:52:11.212 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:52:11.212 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:52:11.212 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:52:11.212 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:11.212 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:11.212 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:11.212 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:52:11.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:52:11.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:11.300 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:11.301 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:52:11.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:52:11.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:52:11.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:360: test_kill_daemons: run_mgr td/ceph-helpers x 2024-09-10T17:52:11.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:52:11.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:52:11.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:52:11.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:52:11.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:52:11.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:52:12.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:52:12.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:12.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:12.222 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:12.222 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:12.222 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:12.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:52:12.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:52:12.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:52:12.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:361: test_kill_daemons: run_osd td/ceph-helpers 0 2024-09-10T17:52:12.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:52:12.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:52:12.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:52:12.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:52:12.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:52:12.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:12.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:52:12.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:52:12.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:52:12.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=56b64570-be8b-46d5-83d4-a84471cf5e9e 2024-09-10T17:52:12.257 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 56b64570-be8b-46d5-83d4-a84471cf5e9e 2024-09-10T17:52:12.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 56b64570-be8b-46d5-83d4-a84471cf5e9e' 2024-09-10T17:52:12.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:52:12.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBMh+BmgRBHEBAAItrKm2gPLn23jrwMSezTbQ== 2024-09-10T17:52:12.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBMh+BmgRBHEBAAItrKm2gPLn23jrwMSezTbQ=="}' 2024-09-10T17:52:12.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 56b64570-be8b-46d5-83d4-a84471cf5e9e -i td/ceph-helpers/0/new.json 2024-09-10T17:52:12.542 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:52:12.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:52:12.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMh+BmgRBHEBAAItrKm2gPLn23jrwMSezTbQ== --osd-uuid 56b64570-be8b-46d5-83d4-a84471cf5e9e 2024-09-10T17:52:12.582 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:12.581+0000 7fb8d7754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:12.611 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:12.610+0000 7fb8d7754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:12.612 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:12.611+0000 7fb8d7754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:12.612 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:12.611+0000 7fb8d7754640 -1 bdev(0x5625be10dc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:52:12.612 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:12.611+0000 7fb8d7754640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:52:18.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:52:18.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:52:18.343 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:52:18.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:52:18.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:52:18.883 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:52:18.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:52:18.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:52:18.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:52:18.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:52:18.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:52:18.910 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:18.909+0000 7fab36633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:18.924 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:18.923+0000 7fab36633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:18.925 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:18.924+0000 7fab36633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:52:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:52:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:52:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:52:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:52:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:52:19.303 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:52:19.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:19.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:19.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:20.298 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:20.297+0000 7fab36633640 -1 Falling back to public interface 2024-09-10T17:52:20.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:20.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:20.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:52:20.732 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:52:20.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:20.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:21.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:22.101 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:22.100+0000 7fab36633640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:52:22.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:22.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:22.149 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:52:22.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:52:22.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:22.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:22.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:23.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:23.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:23.697 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:52:23.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:52:23.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:23.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:24.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:25.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:25.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:25.187 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:52:25.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:52:25.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:25.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:25.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:26.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:26.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:26.586 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:52:26.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:52:26.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:26.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:27.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:28.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:28.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:28.015 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:52:28.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:52:28.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:28.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:28.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:29.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:29.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:29.432 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:52:29.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:52:29.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:29.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:29.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:30.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:30.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:30.935 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:52:30.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:52:30.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:30.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:31.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:32.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:32.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:32.365 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:52:32.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:52:32.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:32.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:32.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:33.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:33.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:33.823 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:52:33.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:52:33.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:33.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:34.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:35.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:35.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:35.583 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:52:35.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:52:35.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:35.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:36.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:37.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:37.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:37.172 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:52:37.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:52:37.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:37.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:37.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:38.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:38.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:38.888 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:52:38.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:52:38.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:38.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:39.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:40.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:40.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:40.632 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:52:40.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:52:40.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:40.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:41.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:42.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:42.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:42.719 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:52:42.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:52:42.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:42.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:43.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:44.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:44.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:44.531 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:52:44.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:52:44.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:44.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:44.967 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3373950300,v1:127.0.0.1:6803/3373950300] [v2:127.0.0.1:6804/3373950300,v1:127.0.0.1:6805/3373950300] exists,up 56b64570-be8b-46d5-83d4-a84471cf5e9e 2024-09-10T17:52:44.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:52:44.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:52:44.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:52:44.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-09-10T17:52:44.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:44.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:44.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:44.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:44.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:45.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 1 2024-09-10T17:52:45.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:372: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-09-10T17:52:45.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:45.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:45.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:45.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:45.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:46.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:46.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:376: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-09-10T17:52:46.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:46.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:46.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:46.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:46.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:46.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:46.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:380: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-09-10T17:52:46.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:46.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:46.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:46.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:46.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:46.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:46.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: timeout 5 ceph status 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:382: test_kill_daemons: teardown td/ceph-helpers 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:51.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:51.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:51.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:52:51.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:52:51.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:52:51.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:52:51.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:52:51.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:52:51.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:51.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:52:51.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:52:51.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:51.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:52:51.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:52:51.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:52:51.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:52:51.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:51.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:51.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:52:51.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:52:51.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:52:51.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:52:51.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_objectstore_tool td/ceph-helpers 2024-09-10T17:52:51.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: test_objectstore_tool: local dir=td/ceph-helpers 2024-09-10T17:52:51.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: test_objectstore_tool: setup td/ceph-helpers 2024-09-10T17:52:51.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:52:51.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:52:51.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:52:51.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:52:51.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:52:51.723 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:51.723 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:51.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:51.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:51.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:51.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:51.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:52:51.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:52:51.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:52:51.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:52:51.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:52:51.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:52:51.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:51.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:52:51.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:52:51.728 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:51.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:52:51.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:52:51.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:52:51.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:52:51.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:51.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:51.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:52:51.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:52:51.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:52:51.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:52:51.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:52:51.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:51.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:51.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:52:51.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:52:51.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:52:52.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:52:52.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:52.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:52.225 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:52.225 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:52.225 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:52.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:52:52.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:52:52.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:52:52.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:52:52.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:52:52.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:52:52.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:52:52.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:52:52.437 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:52:52.437 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:52:52.438 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:52:52.438 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:52.438 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:52.438 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:52.438 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:52:52.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:52:52.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:52:52.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:52:52.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:52:52.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:52:52.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:52:52.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:52:52.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: test_objectstore_tool: local osd=0 2024-09-10T17:52:52.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-09-10T17:52:52.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:52:52.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:52:52.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:52:52.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:52:52.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:52:52.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:52:53.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:52:53.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:53.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:53.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:53.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:53.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:53.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:52:53.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:52:53.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:52:53.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-09-10T17:52:53.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:52:53.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:52:53.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:52:53.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a617680-e6ab-44fb-b36a-0646a98f99ef 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3a617680-e6ab-44fb-b36a-0646a98f99ef' 2024-09-10T17:52:53.484 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 3a617680-e6ab-44fb-b36a-0646a98f99ef 2024-09-10T17:52:53.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:52:53.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB1h+Bm5lbLHRAA8s3k7zM+noBnqZiGScKWvg== 2024-09-10T17:52:53.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB1h+Bm5lbLHRAA8s3k7zM+noBnqZiGScKWvg=="}' 2024-09-10T17:52:53.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a617680-e6ab-44fb-b36a-0646a98f99ef -i td/ceph-helpers/0/new.json 2024-09-10T17:52:53.776 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:52:53.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:52:53.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1h+Bm5lbLHRAA8s3k7zM+noBnqZiGScKWvg== --osd-uuid 3a617680-e6ab-44fb-b36a-0646a98f99ef 2024-09-10T17:52:53.816 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:53.815+0000 7f5ad9817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:53.828 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:53.827+0000 7f5ad9817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:53.829 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:53.828+0000 7f5ad9817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:53.829 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:53.829+0000 7f5ad9817640 -1 bdev(0x556a220f3c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:52:53.830 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:53.829+0000 7f5ad9817640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:52:59.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:52:59.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:52:59.332 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:52:59.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:52:59.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:52:59.879 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:52:59.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:52:59.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:52:59.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:52:59.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:52:59.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:52:59.906 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:59.905+0000 7f765120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:59.908 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:59.907+0000 7f765120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:59.909 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:52:59.908+0000 7f765120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:00.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:53:00.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:53:00.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:53:00.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:53:00.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:53:00.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:00.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:53:00.304 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:53:00.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:00.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:00.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:00.757 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:00.756+0000 7f765120f640 -1 Falling back to public interface 2024-09-10T17:53:01.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:01.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:01.772 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:53:01.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:53:01.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:01.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:02.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:02.508 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:02.507+0000 7f765120f640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:53:03.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:03.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:03.129 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:53:03.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:53:03.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:03.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:03.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:04.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:04.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:04.677 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:53:04.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:53:04.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:04.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:05.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:06.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:06.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:06.085 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:53:06.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:53:06.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:06.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:06.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:07.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:07.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:07.485 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:53:07.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:53:07.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:07.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:07.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:08.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:08.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:08.904 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:53:08.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:53:08.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:08.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:09.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:10.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:10.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:10.409 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:53:10.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:53:10.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:10.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:10.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:11.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:11.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:11.802 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:53:11.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:53:11.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:11.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:12.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:13.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:13.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:13.264 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:53:13.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:53:13.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:13.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:13.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:14.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:14.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:14.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:53:14.679 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:53:14.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:14.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:15.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:16.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:16.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:16.180 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:53:16.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:53:16.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:16.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:16.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:17.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:17.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:17.698 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:53:17.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:53:17.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:17.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:18.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:19.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:19.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:19.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:53:19.670 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:53:19.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:19.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:20.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:21.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:21.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:21.555 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:53:21.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:53:21.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:21.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:22.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:23.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:23.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:23.515 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:53:23.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:53:23.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:23.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:24.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:25.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:25.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:25.112 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:53:25.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:53:25.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:25.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:25.510 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2543876389,v1:127.0.0.1:6803/2543876389] [v2:127.0.0.1:6804/2543876389,v1:127.0.0.1:6805/2543876389] exists,up 3a617680-e6ab-44fb-b36a-0646a98f99ef 2024-09-10T17:53:25.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:53:25.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:53:25.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:53:25.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: test_objectstore_tool: create_rbd_pool 2024-09-10T17:53:25.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:53:25.920 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:53:25.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:53:25.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:53:26.622 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:53:26.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:53:27.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:53:28.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: test_objectstore_tool: wait_for_clean 2024-09-10T17:53:28.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:53:28.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:53:28.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:53:28.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:53:28.812 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:53:28.812 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:53:28.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:53:28.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:53:28.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:53:28.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:53:28.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:53:28.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:53:28.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:53:28.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:53:28.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:53:29.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:53:29.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:53:29.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:53:29.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:53:29.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836485 2024-09-10T17:53:29.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836485 2024-09-10T17:53:29.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836485' 2024-09-10T17:53:29.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:53:29.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836485 2024-09-10T17:53:29.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:53:29.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:53:29.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836485 2024-09-10T17:53:29.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:53:29.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836485 2024-09-10T17:53:29.410 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836485 2024-09-10T17:53:29.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-10T17:53:29.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:53:29.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-10T17:53:29.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:53:30.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:53:30.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:53:31.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-10T17:53:31.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:53:32.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T17:53:32.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:53:32.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-10T17:53:32.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:53:32.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:53:32.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:53:33.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:53:33.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:53:33.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:53:33.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:53:33.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:53:33.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:53:33.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:53:33.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:53:33.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:53:33.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:53:33.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:53:33.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:53:34.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:53:34.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:53:34.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:53:34.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-09-10T17:53:34.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: diff - /etc/group 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T17:53:34.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:53:34.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-10T17:53:34.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:53:34.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:53:34.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:53:34.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:53:34.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:53:34.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:53:34.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-09-10T17:53:34.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-10T17:53:34.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:53:34.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T17:53:34.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:53:34.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-10T17:53:34.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:53:35.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:53:35.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:start osd.0 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:53:35.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-10T17:53:35.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T17:53:35.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:53:35.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:53:35.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:53:35.260 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:35.259+0000 7f3837dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:35.262 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:35.261+0000 7f3837dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:35.263 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:35.262+0000 7f3837dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:0 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:35.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:36.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:36.613 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:36.612+0000 7f3837dc5640 -1 Falling back to public interface 2024-09-10T17:53:37.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:37.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:37.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:53:37.070 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-10T17:53:37.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:37.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:37.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:38.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:38.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:38.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:53:38.528 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-10T17:53:38.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:38.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:38.741 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:38.741+0000 7f3837dc5640 -1 osd.0 14 log_to_monitors true 2024-09-10T17:53:38.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:40.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:40.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:40.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:53:40.001 INFO:tasks.workunit.client.0.smithi155.stderr:3 2024-09-10T17:53:40.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:40.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:40.422 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 up in weight 1 up_from 17 up_thru 17 down_at 15 last_clean_interval [5,14) [v2:127.0.0.1:6802/2840642456,v1:127.0.0.1:6803/2840642456] [v2:127.0.0.1:6804/2840642456,v1:127.0.0.1:6805/2840642456] exists,up 3a617680-e6ab-44fb-b36a-0646a98f99ef 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:53:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:53:40.424 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:53:40.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:53:40.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:53:40.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:53:40.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:53:40.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:53:40.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:53:40.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:53:40.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:53:40.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:53:40.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:53:40.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:53:40.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:53:40.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:53:41.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=73014444035 2024-09-10T17:53:41.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 73014444035 2024-09-10T17:53:41.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-73014444035' 2024-09-10T17:53:41.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:53:41.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-73014444035 2024-09-10T17:53:41.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:53:41.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:53:41.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-73014444035 2024-09-10T17:53:41.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:53:41.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=73014444035 2024-09-10T17:53:41.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 73014444035' 2024-09-10T17:53:41.047 INFO:tasks.workunit.client.0.smithi155.stderr:waiting osd.0 seq 73014444035 2024-09-10T17:53:41.047 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:53:41.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 73014444035 2024-09-10T17:53:41.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:53:42.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:53:42.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:53:42.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 73014444035 -lt 73014444035 2024-09-10T17:53:42.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:53:42.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:53:42.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:53:43.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:53:43.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:53:43.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:53:43.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:53:43.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:53:43.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:53:43.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:53:43.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:53:43.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:53:43.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:53:43.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:53:43.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:53:44.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T17:53:44.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:53:44.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-10T17:53:44.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:53:44.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:53:44.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:53:44.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:53:44.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:53:44.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:53:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-10T17:53:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-10T17:53:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:53:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T17:53:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:53:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-10T17:53:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-09-10T17:53:44.994 INFO:tasks.workunit.client.0.smithi155.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-09-10T17:53:45.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-10T17:53:45.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: return 1 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: return 1 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: teardown td/ceph-helpers 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:53:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:53:45.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:53:45.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:53:45.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:53:45.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:53:45.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:53:45.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:53:45.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:53:45.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:53:45.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:53:45.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:53:45.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:53:45.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:53:45.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:53:45.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:53:45.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:53:45.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:45.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:45.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_pg_schedule_scrub td/ceph-helpers 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: test_pg_schedule_scrub: local dir=td/ceph-helpers 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_schedule_scrub: setup td/ceph-helpers 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:53:45.381 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:53:45.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:53:45.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:53:45.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:53:45.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:53:45.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:53:45.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:53:45.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:53:45.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:53:45.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:53:45.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:53:45.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:53:45.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:53:45.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:53:45.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:53:45.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:53:45.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:53:45.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:53:45.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:53:45.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:45.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:45.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:53:45.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:53:45.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:53:45.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:53:45.391 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:53:45.391 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:45.391 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:45.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:53:45.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:53:45.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:53:45.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:53:45.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:53:45.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:53:45.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:53:45.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:53:45.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:53:45.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:53:45.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:53:45.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:53:45.848 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:53:45.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:53:45.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:53:45.849 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:53:45.849 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:45.849 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:45.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:53:45.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:53:46.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:53:46.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:53:46.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:53:46.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:53:46.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:53:46.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:53:46.042 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:53:46.042 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:53:46.042 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:53:46.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:53:46.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:46.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:46.043 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:53:46.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:53:46.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:53:46.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:53:46.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:53:46.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:53:46.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:53:46.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:53:46.125 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:53:46.126 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:53:46.126 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:53:46.126 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:53:46.126 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:46.126 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:46.126 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:53:46.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:53:46.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:53:46.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: test_pg_schedule_scrub: run_mgr td/ceph-helpers x 2024-09-10T17:53:46.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:53:46.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:53:46.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:53:46.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:53:46.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:53:46.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:53:47.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:53:47.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:53:47.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:53:47.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:53:47.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:47.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:47.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:53:47.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:53:47.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:53:47.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1944: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2024-09-10T17:53:47.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:53:47.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:53:47.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:53:47.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ef613c92-895f-49ff-942d-9425782dea4d 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ef613c92-895f-49ff-942d-9425782dea4d' 2024-09-10T17:53:47.091 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 ef613c92-895f-49ff-942d-9425782dea4d 2024-09-10T17:53:47.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:53:47.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCrh+BmPf9aBhAANZhIuf7ayPPyDN9wDfmtTw== 2024-09-10T17:53:47.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCrh+BmPf9aBhAANZhIuf7ayPPyDN9wDfmtTw=="}' 2024-09-10T17:53:47.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ef613c92-895f-49ff-942d-9425782dea4d -i td/ceph-helpers/0/new.json 2024-09-10T17:53:47.373 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:53:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:53:47.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrh+BmPf9aBhAANZhIuf7ayPPyDN9wDfmtTw== --osd-uuid ef613c92-895f-49ff-942d-9425782dea4d 2024-09-10T17:53:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:47.412+0000 7f6f6ff54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:47.434 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:47.433+0000 7f6f6ff54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:47.435 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:47.434+0000 7f6f6ff54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:47.435 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:47.434+0000 7f6f6ff54640 -1 bdev(0x55f58dda9c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:53:47.435 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:47.434+0000 7f6f6ff54640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:53:53.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:53:53.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:53:53.420 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:53:53.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:53:53.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:53:53.973 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:53:53.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:53:53.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:53:53.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:53:53.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:53:53.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:53:54.000 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:53.999+0000 7f1a2deb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:54.003 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:54.002+0000 7f1a2deb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:54.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:54.003+0000 7f1a2deb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:54.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:53:54.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:53:54.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:53:54.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:53:54.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:53:54.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:54.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:53:54.409 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:53:54.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:54.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:54.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:55.600 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:55.599+0000 7f1a2deb6640 -1 Falling back to public interface 2024-09-10T17:53:55.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:55.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:55.801 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:53:55.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:53:55.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:55.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:56.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:57.180 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:53:57.179+0000 7f1a2deb6640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:53:57.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:57.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:57.229 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:53:57.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:53:57.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:57.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:57.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:58.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:58.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:58.784 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:53:58.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:53:58.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:58.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:53:59.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:00.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:00.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:00.238 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:54:00.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:54:00.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:00.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:00.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:01.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:01.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:01.653 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:54:01.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:54:01.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:01.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:02.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:03.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:03.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:03.077 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:54:03.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:54:03.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:03.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:03.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:04.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:04.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:04.474 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:54:04.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:54:04.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:04.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:04.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:05.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:05.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:05.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:54:05.912 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:54:05.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:05.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:06.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:07.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:07.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:07.332 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:54:07.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:54:07.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:07.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:07.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:08.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:08.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:08.737 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:54:08.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:54:08.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:08.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:09.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:10.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:10.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:10.442 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:54:10.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:54:10.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:10.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:11.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:12.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:12.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:12.028 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:54:12.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:54:12.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:12.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:12.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:13.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:13.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:13.777 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:54:13.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:54:13.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:13.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:14.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:15.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:15.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:15.808 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:54:15.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:54:15.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:15.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:16.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:17.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:17.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:17.854 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:54:17.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:54:17.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:17.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:19.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:20.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:20.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:20.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:54:20.092 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:54:20.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:20.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:20.486 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3555040062,v1:127.0.0.1:6803/3555040062] [v2:127.0.0.1:6804/3555040062,v1:127.0.0.1:6805/3555040062] exists,up ef613c92-895f-49ff-942d-9425782dea4d 2024-09-10T17:54:20.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:54:20.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:54:20.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:54:20.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: test_pg_schedule_scrub: create_rbd_pool 2024-09-10T17:54:20.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:54:20.863 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:54:20.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:54:20.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:54:21.460 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:54:21.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:54:22.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:54:23.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: test_pg_schedule_scrub: wait_for_clean 2024-09-10T17:54:23.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:54:23.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:54:23.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:54:23.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:54:23.431 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:54:23.431 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:54:23.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:54:23.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:54:23.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:54:23.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:54:23.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:54:23.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:54:23.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:54:23.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:54:23.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:54:23.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:54:23.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:54:23.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:54:23.943 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:54:24.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-10T17:54:24.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-10T17:54:24.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-10T17:54:24.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:54:24.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-10T17:54:24.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:54:24.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:54:24.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-10T17:54:24.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:54:24.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-10T17:54:24.060 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836488 2024-09-10T17:54:24.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-10T17:54:24.061 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:54:24.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-10T17:54:24.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:54:25.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:54:25.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:54:25.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-10T17:54:25.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:54:25.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:54:25.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:54:26.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:54:26.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:54:26.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:54:26.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:54:26.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:54:26.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:54:26.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:54:26.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:54:26.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:54:26.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:54:26.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:54:26.857 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:54:27.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:54:27.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:54:27.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:54:27.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2024-09-10T17:54:27.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_schedule_scrub: local pgid=1.0 2024-09-10T17:54:27.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_schedule_scrub: get_last_scrub_stamp 1.0 2024-09-10T17:54:27.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:54:27.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:54:27.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:54:27.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:54:27.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_schedule_scrub: local last_scrub=2024-09-10T17:54:21.301827+0000 2024-09-10T17:54:27.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_schedule_scrub: ceph pg scrub 1.0 2024-09-10T17:54:28.143 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-10T17:54:28.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_schedule_scrub: wait_for_scrub 1.0 2024-09-10T17:54:21.301827+0000 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T17:54:21.301827+0000 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:54:28.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:54:28.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:54:21.301827+0000 '>' 2024-09-10T17:54:21.301827+0000 2024-09-10T17:54:28.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:54:29.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:54:29.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:54:29.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:54:29.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:54:29.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:54:29.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:54:29.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:54:29.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:54:21.301827+0000 '>' 2024-09-10T17:54:21.301827+0000 2024-09-10T17:54:29.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:54:30.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:54:30.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:54:30.928 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:54:30.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:54:30.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:54:30.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:54:30.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:54:31.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:54:29.105975+0000 '>' 2024-09-10T17:54:21.301827+0000 2024-09-10T17:54:31.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T17:54:31.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1948: test_pg_schedule_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-10T17:54:31.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:54:31.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:54:31.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:54:31.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:54:31.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:54:31.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:54:31.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_schedule_scrub: TIMEOUT=1 2024-09-10T17:54:31.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_schedule_scrub: pg_scrub 1.0 2024-09-10T17:54:31.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-10T17:54:31.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-10T17:54:31.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:54:31.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:54:31.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:54:31.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:54:31.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T17:54:29.105975+0000 2024-09-10T17:54:31.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-10T17:54:32.163 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-10T17:54:29.105975+0000 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T17:54:29.105975+0000 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:54:32.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:54:32.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:54:29.105975+0000 '>' 2024-09-10T17:54:29.105975+0000 2024-09-10T17:54:32.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:54:33.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:54:33.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: return 1 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_schedule_scrub: teardown td/ceph-helpers 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:54:33.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:54:33.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:54:33.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:54:33.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:54:33.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:54:33.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:54:33.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:54:33.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:54:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:54:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:54:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:54:33.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:54:33.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:54:33.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:54:33.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:54:33.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:54:33.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:33.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:54:33.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:54:33.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:54:33.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:54:33.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:54:33.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_pg_scrub td/ceph-helpers 2024-09-10T17:54:33.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: test_pg_scrub: local dir=td/ceph-helpers 2024-09-10T17:54:33.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: test_pg_scrub: setup td/ceph-helpers 2024-09-10T17:54:33.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:54:33.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:54:33.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:54:33.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:54:33.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:54:33.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:54:33.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:54:33.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:54:33.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:54:33.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:54:33.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:54:33.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:54:33.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:54:33.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:54:33.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:54:33.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:54:33.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:54:33.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:54:33.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:54:33.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:54:33.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:54:33.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:54:33.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:54:33.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:54:33.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:54:33.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:33.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:54:33.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:54:33.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:54:33.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:54:33.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:54:33.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:54:33.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:33.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:54:33.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:54:33.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:54:33.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:54:33.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:54:33.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:54:33.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:54:33.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:54:33.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:54:33.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:54:33.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:54:33.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:54:33.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:54:34.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:54:34.151 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:54:34.151 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:54:34.151 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:54:34.151 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:34.151 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:54:34.151 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:54:34.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:54:34.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:54:34.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:54:34.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:54:34.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:54:34.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:54:34.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:54:34.356 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:54:34.356 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:54:34.356 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:54:34.356 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:54:34.357 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:34.357 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:54:34.357 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:54:34.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:54:34.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:54:34.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:54:34.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:54:34.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:54:34.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:54:34.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:54:34.445 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:54:34.445 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:54:34.445 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:54:34.445 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:54:34.445 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:34.445 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:54:34.446 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:54:34.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:54:34.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:54:34.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: test_pg_scrub: run_mgr td/ceph-helpers x 2024-09-10T17:54:34.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:54:34.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:54:34.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:54:34.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:54:34.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:54:34.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:54:35.373 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:54:35.373 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:54:35.373 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:54:35.373 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:54:35.373 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:35.373 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:54:35.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:54:35.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:54:35.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:54:35.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1902: test_pg_scrub: run_osd td/ceph-helpers 0 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:54:35.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:54:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:54:35.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:54:35.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2b81a5e0-895c-4bd0-8a1a-3a83c6e7146d 2024-09-10T17:54:35.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2b81a5e0-895c-4bd0-8a1a-3a83c6e7146d' 2024-09-10T17:54:35.408 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 2b81a5e0-895c-4bd0-8a1a-3a83c6e7146d 2024-09-10T17:54:35.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:54:35.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbh+BmEtc3GRAAVwPMWRW7eAGe6xGOm7Cn5Q== 2024-09-10T17:54:35.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbh+BmEtc3GRAAVwPMWRW7eAGe6xGOm7Cn5Q=="}' 2024-09-10T17:54:35.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2b81a5e0-895c-4bd0-8a1a-3a83c6e7146d -i td/ceph-helpers/0/new.json 2024-09-10T17:54:35.691 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:54:35.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:54:35.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbh+BmEtc3GRAAVwPMWRW7eAGe6xGOm7Cn5Q== --osd-uuid 2b81a5e0-895c-4bd0-8a1a-3a83c6e7146d 2024-09-10T17:54:35.731 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:35.730+0000 7f4ef7cd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:35.760 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:35.759+0000 7f4ef7cd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:35.761 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:35.760+0000 7f4ef7cd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:35.762 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:35.760+0000 7f4ef7cd4640 -1 bdev(0x561745e27c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:54:35.762 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:35.761+0000 7f4ef7cd4640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:54:41.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:54:41.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:54:41.539 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:54:41.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:54:41.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:54:42.075 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:54:42.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:54:42.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:54:42.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:54:42.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:54:42.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:54:42.103 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:42.102+0000 7fcb74a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:42.121 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:42.120+0000 7fcb74a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:42.122 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:42.121+0000 7fcb74a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:42.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:54:42.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:54:42.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:54:42.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:54:42.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:54:42.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:42.535 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:54:42.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:54:42.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:42.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:42.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:43.721 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:43.720+0000 7fcb74a15640 -1 Falling back to public interface 2024-09-10T17:54:43.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:43.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:43.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:54:43.954 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:54:43.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:43.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:44.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:45.290 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:54:45.289+0000 7fcb74a15640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:54:45.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:45.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:45.388 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:54:45.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:54:45.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:45.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:45.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:46.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:46.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:46.919 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:54:46.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:54:46.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:46.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:47.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:48.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:48.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:48.391 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:54:48.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:54:48.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:48.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:48.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:49.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:49.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:49.785 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:54:49.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:54:49.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:49.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:50.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:51.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:51.187 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:54:51.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:51.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:54:51.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:51.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:51.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:52.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:52.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:52.621 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:54:52.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:54:52.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:52.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:53.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:54.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:54.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:54.075 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:54:54.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:54:54.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:54.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:54.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:55.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:55.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:55.498 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:54:55.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:54:55.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:55.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:55.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:56.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:56.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:56.901 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:54:56.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:54:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:57.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:58.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:58.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:58.311 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:54:58.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:54:58.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:58.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:54:59.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:00.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:00.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:00.035 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:55:00.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:55:00.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:00.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:00.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:01.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:01.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:01.788 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:55:01.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:55:01.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:01.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:02.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:03.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:03.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:03.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:55:03.984 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:55:03.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:03.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:04.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:05.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:05.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:05.854 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:55:05.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:55:05.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:05.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:06.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:07.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:07.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:07.505 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:55:07.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:55:07.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:07.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:07.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:08.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:08.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:08.973 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:55:08.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:55:08.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:08.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:09.391 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1272618830,v1:127.0.0.1:6803/1272618830] [v2:127.0.0.1:6804/1272618830,v1:127.0.0.1:6805/1272618830] exists,up 2b81a5e0-895c-4bd0-8a1a-3a83c6e7146d 2024-09-10T17:55:09.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:55:09.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:55:09.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:55:09.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: test_pg_scrub: create_rbd_pool 2024-09-10T17:55:09.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:55:09.789 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:55:09.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:55:09.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:55:10.306 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:55:10.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:55:11.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:55:12.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1904: test_pg_scrub: wait_for_clean 2024-09-10T17:55:12.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:55:12.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:55:12.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:55:12.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:55:12.447 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:55:12.447 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:55:12.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:55:12.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:55:12.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:55:12.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:55:12.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:55:12.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:55:12.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:55:12.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:55:12.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:55:12.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:55:12.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:55:12.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:55:12.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:55:13.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-10T17:55:13.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-10T17:55:13.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-10T17:55:13.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:55:13.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-10T17:55:13.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:55:13.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:55:13.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-10T17:55:13.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:55:13.081 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836488 2024-09-10T17:55:13.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-10T17:55:13.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-10T17:55:13.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:55:13.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-10T17:55:13.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:55:13.473 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:55:13.473 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:55:13.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:55:13.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:55:13.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:55:13.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:55:13.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:55:13.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:55:13.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:55:13.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:55:14.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:55:14.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:55:14.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:55:14.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1905: test_pg_scrub: pg_scrub 1.0 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:55:14.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:55:15.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T17:55:10.125896+0000 2024-09-10T17:55:15.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-10T17:55:15.712 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-10T17:55:10.125896+0000 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T17:55:10.125896+0000 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:55:15.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:55:16.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:55:10.125896+0000 '>' 2024-09-10T17:55:10.125896+0000 2024-09-10T17:55:16.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:55:17.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:55:17.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:55:17.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:55:17.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:55:17.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:55:17.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:55:17.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:55:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:55:10.125896+0000 '>' 2024-09-10T17:55:10.125896+0000 2024-09-10T17:55:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:55:18.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:55:18.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:55:18.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:55:18.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:55:18.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:55:18.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:55:18.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:55:18.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:55:10.125896+0000 '>' 2024-09-10T17:55:10.125896+0000 2024-09-10T17:55:18.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:55:19.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:55:19.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:55:19.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:55:19.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:55:19.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:55:19.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:55:19.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:55:20.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:55:16.273135+0000 '>' 2024-09-10T17:55:10.125896+0000 2024-09-10T17:55:20.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T17:55:20.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-10T17:55:20.273 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:55:20.273 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:55:20.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:55:20.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:55:20.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_pg_scrub: TIMEOUT=1 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_pg_scrub: pg_scrub 1.0 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:55:20.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:55:20.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T17:55:16.273135+0000 2024-09-10T17:55:20.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-10T17:55:21.166 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-10T17:55:16.273135+0000 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T17:55:16.273135+0000 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:55:21.178 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:55:21.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:55:16.273135+0000 '>' 2024-09-10T17:55:16.273135+0000 2024-09-10T17:55:21.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:55:22.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:55:22.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: return 1 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: test_pg_scrub: teardown td/ceph-helpers 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:55:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:55:22.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:55:22.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:55:22.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:55:22.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:55:22.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:55:22.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:55:22.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:55:22.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:55:22.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:55:22.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:55:22.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:55:22.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:55:22.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:55:22.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:55:22.677 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:55:22.678 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:22.678 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:55:22.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_repair td/ceph-helpers 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1854: test_repair: local dir=td/ceph-helpers 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_repair: setup td/ceph-helpers 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:55:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:55:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:55:22.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:55:22.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:55:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:55:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:55:22.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:55:22.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:55:22.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:55:22.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:55:22.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:55:22.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:55:22.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:55:22.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:55:22.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:55:22.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:55:22.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:55:22.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:55:22.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:55:22.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:55:22.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:55:22.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:55:22.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:22.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:55:22.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:55:22.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:55:22.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:55:22.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-10T17:55:22.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:55:22.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:22.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:55:22.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.76675 2024-09-10T17:55:22.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:55:22.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:55:22.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:55:22.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-10T17:55:22.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:55:22.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-10T17:55:22.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:55:22.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:55:22.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:55:22.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-10T17:55:22.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:55:23.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:55:23.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:55:23.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:55:23.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:55:23.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:23.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:55:23.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:55:23.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T17:55:23.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:55:23.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:55:23.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:55:23.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:55:23.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:55:23.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:55:23.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get fsid 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:23.445 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:55:23.446 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.76675/ceph-mon.a.asok 2024-09-10T17:55:23.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:55:23.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.76675/ceph-mon.a.asok config get mon_host 2024-09-10T17:55:23.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: test_repair: run_mgr td/ceph-helpers x 2024-09-10T17:55:23.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-10T17:55:23.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:55:23.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:55:23.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:55:23.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-10T17:55:23.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:55:24.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:55:24.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:55:24.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:55:24.377 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:55:24.377 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:24.377 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:55:24.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:55:24.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:55:24.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:55:24.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1859: test_repair: run_osd td/ceph-helpers 0 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-10T17:55:24.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:55:24.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-10T17:55:24.411 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:55:24.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=251e86fa-431c-4111-a860-d74ba326b7a6 2024-09-10T17:55:24.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 251e86fa-431c-4111-a860-d74ba326b7a6' 2024-09-10T17:55:24.412 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 251e86fa-431c-4111-a860-d74ba326b7a6 2024-09-10T17:55:24.413 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:55:24.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAMiOBmMtF3GRAAhtVpI1HznanmO9iam27w4Q== 2024-09-10T17:55:24.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAMiOBmMtF3GRAAhtVpI1HznanmO9iam27w4Q=="}' 2024-09-10T17:55:24.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 251e86fa-431c-4111-a860-d74ba326b7a6 -i td/ceph-helpers/0/new.json 2024-09-10T17:55:24.695 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:55:24.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-10T17:55:24.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMiOBmMtF3GRAAhtVpI1HznanmO9iam27w4Q== --osd-uuid 251e86fa-431c-4111-a860-d74ba326b7a6 2024-09-10T17:55:24.735 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:24.733+0000 7f877dc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:55:24.764 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:24.763+0000 7f877dc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:55:24.765 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:24.764+0000 7f877dc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:55:24.765 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:24.764+0000 7f877dc0d640 -1 bdev(0x556b94b3fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:55:24.765 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:24.764+0000 7f877dc0d640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-10T17:55:30.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-10T17:55:30.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:55:30.261 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-10T17:55:30.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:55:30.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:55:30.803 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-10T17:55:30.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:55:30.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d8847490-5108-4323-8696-d589aafcc558 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76675/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:55:30.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:55:30.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:55:30.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:55:30.831 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:30.829+0000 7f4b97e87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:55:30.841 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:30.840+0000 7f4b97e87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:55:30.842 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:30.841+0000 7f4b97e87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:55:31.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:55:31.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:55:31.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:55:31.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:55:31.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:55:31.256 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-10T17:55:31.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:31.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:55:31.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:31.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:31.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:32.439 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:32.438+0000 7f4b97e87640 -1 Falling back to public interface 2024-09-10T17:55:32.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:32.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:32.674 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-10T17:55:32.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:55:32.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:32.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:33.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:34.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:34.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:34.126 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-10T17:55:34.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:55:34.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:34.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:34.251 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-10T17:55:34.249+0000 7f4b97e87640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:55:34.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:35.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:35.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:35.544 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-10T17:55:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:55:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:36.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:37.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:37.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:37.009 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-10T17:55:37.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:55:37.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:37.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:37.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:38.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:38.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:38.438 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-10T17:55:38.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:55:38.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:38.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:38.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:39.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:39.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:39.836 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-10T17:55:39.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:55:39.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:39.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:40.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:41.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:41.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:41.340 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-10T17:55:41.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:55:41.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:41.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:41.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:42.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:42.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:42.762 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-10T17:55:42.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:55:42.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:42.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:43.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:44.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:44.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:44.155 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-10T17:55:44.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:55:44.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:44.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:44.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:45.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:45.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:45.551 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-10T17:55:45.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:55:45.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:45.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:46.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:47.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:47.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:47.074 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-10T17:55:47.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:55:47.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:47.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:48.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:48.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:48.751 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-10T17:55:48.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:55:48.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:48.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:49.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:50.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:50.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:50.506 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-10T17:55:50.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:55:50.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:50.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:50.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:51.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:51.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:51.925 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-10T17:55:51.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:55:51.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:51.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:52.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:53.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:53.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:53.626 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-10T17:55:53.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:55:53.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:53.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:54.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:55.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:55.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:55.429 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-10T17:55:55.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:55:55.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:55.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:56.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:57.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:57.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:57.214 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-10T17:55:57.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:55:57.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:57.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:55:57.660 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2084250626,v1:127.0.0.1:6803/2084250626] [v2:127.0.0.1:6804/2084250626,v1:127.0.0.1:6805/2084250626] exists,up 251e86fa-431c-4111-a860-d74ba326b7a6 2024-09-10T17:55:57.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:55:57.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:55:57.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:55:57.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1860: test_repair: create_rbd_pool 2024-09-10T17:55:57.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T17:55:58.037 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-10T17:55:58.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T17:55:58.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:55:58.573 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-10T17:55:58.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:55:59.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T17:56:00.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1861: test_repair: wait_for_clean 2024-09-10T17:56:01.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:56:01.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:56:01.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:56:01.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:56:01.005 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:56:01.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:56:01.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T17:56:01.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:56:01.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:56:01.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:56:01.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-10T17:56:01.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-10T17:56:01.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-10T17:56:01.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:56:01.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-10T17:56:01.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:56:01.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:56:01.391 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-10T17:56:01.391 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:56:01.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-10T17:56:01.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-10T17:56:01.392 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836488 2024-09-10T17:56:01.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:56:01.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-10T17:56:01.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:56:02.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:56:02.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:56:03.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-10T17:56:03.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:56:03.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:56:03.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:56:03.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T17:56:03.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:56:03.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:56:03.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:56:03.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:56:03.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:56:03.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:56:03.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:56:04.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:56:04.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:56:04.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:56:04.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:56:04.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T17:56:04.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:56:04.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:56:04.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1862: test_repair: repair 1.0 2024-09-10T17:56:04.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-10T17:56:04.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-10T17:56:04.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:56:04.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:56:04.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:56:04.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:56:05.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T17:55:58.391946+0000 2024-09-10T17:56:05.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-10T17:56:05.510 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-10T17:56:05.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-10T17:55:58.391946+0000 2024-09-10T17:56:05.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T17:56:05.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T17:55:58.391946+0000 2024-09-10T17:56:05.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T17:56:05.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T17:56:05.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:56:05.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:56:05.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:56:05.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:56:05.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:56:05.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:56:05.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:55:58.391946+0000 '>' 2024-09-10T17:55:58.391946+0000 2024-09-10T17:56:05.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:56:06.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:56:06.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:56:06.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:56:06.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:56:06.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:56:06.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:56:06.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:56:07.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:55:58.391946+0000 '>' 2024-09-10T17:55:58.391946+0000 2024-09-10T17:56:07.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:56:08.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:56:08.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T17:56:08.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:56:08.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:56:08.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:56:08.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:56:08.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:56:08.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:56:06.364495+0000 '>' 2024-09-10T17:55:58.391946+0000 2024-09-10T17:56:08.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T17:56:08.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T17:56:10.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1863: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-09-10T17:56:10.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:56:10.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:56:10.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:56:10.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:56:10.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:56:10.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:56:10.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: TIMEOUT=1 2024-09-10T17:56:10.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: repair 1.0 2024-09-10T17:56:10.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-10T17:56:10.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-10T17:56:10.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:56:10.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:56:10.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:56:10.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:56:11.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T17:56:06.364495+0000 2024-09-10T17:56:11.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-10T17:56:11.547 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-10T17:56:11.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-10T17:56:06.364495+0000 2024-09-10T17:56:11.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T17:56:11.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T17:56:06.364495+0000 2024-09-10T17:56:11.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T17:56:11.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T17:56:11.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-10T17:56:11.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T17:56:11.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T17:56:11.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T17:56:11.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T17:56:11.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T17:56:11.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T17:56:06.364495+0000 '>' 2024-09-10T17:56:06.364495+0000 2024-09-10T17:56:11.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T17:56:12.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T17:56:12.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-10T17:56:12.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: return 1 2024-09-10T17:56:12.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T17:56:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: return 1 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: run_tests: teardown td/ceph-helpers 1 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:56:14.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:56:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:56:15.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:56:15.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:56:15.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:56:15.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:56:15.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:56:15.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:56:15.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:56:15.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:56:15.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:56:15.049 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:56:15.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:56:15.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-10T17:56:15.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-10T17:56:15.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-10T17:56:15.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log /home/ubuntu/cephtest/archive/log 2024-09-10T17:56:15.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-10T17:56:15.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:56:15.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:56:15.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.76675 2024-09-10T17:56:15.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.76675 2024-09-10T17:56:15.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:56:15.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:56:15.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: run_tests: return 1 2024-09-10T17:56:15.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: : exit 1 2024-09-10T17:56:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:56:15.061 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-09-10T17:56:15.062 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T17:56:15.359 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-09-10T17:56:15.558 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8a2cec67d81c491abac9c36958ccfa9c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-09-10T17:56:15.560 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T17:56:15.569 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T17:56:15.569 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T17:56:15.625 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T17:56:15.633 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T17:56:15.633 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T17:56:15.653 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-09-10T17:56:15.658 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T17:56:15.658 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-09-10T17:56:15.658 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv01.front.sepia.ceph.com 3 6 377 49 +255us[ +237us] +/- 55ms 2024-09-10T17:56:15.658 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 +417us[ +400us] +/- 67ms 2024-09-10T17:56:15.658 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -820us[ -820us] +/- 68ms 2024-09-10T17:56:15.658 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T17:56:15.659 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T17:56:15.667 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T17:56:15.668 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T17:56:15.677 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T17:56:15.758 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T17:56:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:56:15.786 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T17:56:15.795 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T17:56:15.810 INFO:teuthology.task.internal:Duration was 2545.582824 seconds 2024-09-10T17:56:15.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T17:56:15.819 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T17:56:15.820 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T17:56:15.865 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T17:56:16.068 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T17:56:16.068 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-09-10T17:56:16.069 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T17:56:16.131 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T17:56:16.131 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T17:56:16.197 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T17:56:16.198 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T17:56:16.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T17:56:16.347 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T17:56:16.347 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T17:56:16.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T17:56:16.385 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:56:16.437 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-09-10T17:56:16.455 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:56:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:56:16.510 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T17:56:16.520 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T17:56:16.522 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899394/remote/smithi155 2024-09-10T17:56:16.523 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T17:56:17.078 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T17:56:17.078 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T17:56:17.115 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T17:56:17.124 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T17:56:17.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T17:56:17.143 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T17:56:17.143 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T17:56:17.170 INFO:teuthology.orchestra.run.smithi155.stdout: 265351 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 17:56 /home/ubuntu/cephtest 2024-09-10T17:56:17.171 INFO:teuthology.orchestra.run.smithi155.stdout: 405775 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 17:19 /home/ubuntu/cephtest/mnt.0 2024-09-10T17:56:17.171 INFO:teuthology.orchestra.run.smithi155.stdout: 405776 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 17:38 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T17:56:17.171 INFO:teuthology.orchestra.run.smithi155.stdout: 405819 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 17:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T17:56:17.171 INFO:teuthology.orchestra.run.smithi155.stdout: 420643 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 17:56 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-10T17:56:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:56:17.172 INFO:teuthology.orchestra.run.smithi155.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-10T17:56:17.172 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-10T17:56:17.177 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T17:56:17.187 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T17:56:17.201 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi155 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-10T17:56:17.205 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} duration: 2545.582824230194 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on smithi155 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_skanta@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8a2cec67d81c491abac9c36958ccfa9c status: fail success: false 2024-09-10T17:56:17.205 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T17:56:17.300 INFO:teuthology.run:FAIL